You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2016/06/03 04:22:19 UTC

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 624 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/624/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue

Error Message:
Error message from server should have the name of the down shard

Stack Trace:
java.lang.AssertionError: Error message from server should have the name of the down shard
	at __randomizedtesting.SeedInfo.seed([CFC46D10AFC7D504:FD14CC1BAA65B465]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue(TestDownShardTolerantSearch.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11990 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestDownShardTolerantSearch
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/init-core-data-001
   [junit4]   2> 3238549 INFO  (SUITE-TestDownShardTolerantSearch-seed#[CFC46D10AFC7D504]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 3238550 INFO  (SUITE-TestDownShardTolerantSearch-seed#[CFC46D10AFC7D504]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /yw/vq
   [junit4]   2> 3238552 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3238553 INFO  (Thread-8368) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3238553 INFO  (Thread-8368) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3238654 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ZkTestServer start zk server on port:65284
   [junit4]   2> 3238654 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3238654 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3238657 INFO  (zkCallback-21493-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3092a8bb name:ZooKeeperConnection Watcher:127.0.0.1:65284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3238657 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3238657 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3238657 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 3238660 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3238660 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3238662 INFO  (zkCallback-21494-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@d80cc75 name:ZooKeeperConnection Watcher:127.0.0.1:65284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3238662 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3238662 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3238662 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 3238664 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 3238667 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 3238669 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 3238671 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3238672 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3238675 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3238676 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 3238678 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3238679 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3238681 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3238681 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3238683 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3238684 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3238686 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3238686 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 3238689 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3238689 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3238691 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3238691 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3238693 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3238694 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3238696 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3238696 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3238699 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3238699 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3238951 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1
   [junit4]   2> 3238952 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 3238954 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7d554118{/yw/vq,null,AVAILABLE}
   [junit4]   2> 3238955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.ServerConnector Started ServerConnector@66f116ca{HTTP/1.1,[http/1.1]}{127.0.0.1:56306}
   [junit4]   2> 3238955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.Server Started @3243890ms
   [junit4]   2> 3238955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/tempDir-001/control/data, hostContext=/yw/vq, hostPort=56306, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores}
   [junit4]   2> 3238955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 3238955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001'
   [junit4]   2> 3238955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3238955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 3238956 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3238956 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3238957 INFO  (zkCallback-21495-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7b392de4 name:ZooKeeperConnection Watcher:127.0.0.1:65284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3238957 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3238958 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3238959 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3238959 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/solr.xml
   [junit4]   2> 3238963 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores
   [junit4]   2> 3238964 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.CoreContainer New CoreContainer 379879630
   [junit4]   2> 3238964 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001]
   [junit4]   2> 3238964 WARN  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/lib
   [junit4]   2> 3238964 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 3238965 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 3238966 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3238966 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3238966 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65284/solr
   [junit4]   2> 3238966 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3238966 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3238967 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3238968 INFO  (zkCallback-21498-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5580c64f name:ZooKeeperConnection Watcher:127.0.0.1:65284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3238968 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3238969 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3238970 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$50/1084092672@1468de6e
   [junit4]   2> 3238970 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3238972 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3ca1905 name:ZooKeeperConnection Watcher:127.0.0.1:65284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3238972 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3238973 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 3238976 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 3238981 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 3238984 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 3238986 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 3238989 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 3238991 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 3238993 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 3238995 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 3238996 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3238997 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 3238998 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 3239002 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 3239004 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 3239006 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3239006 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000
   [junit4]   2> 3239007 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56306_yw%2Fvq
   [junit4]   2> 3239007 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 3239009 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.Overseer Overseer (id=96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) starting
   [junit4]   2> 3239012 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 3239020 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 3239021 INFO  (OverseerCollectionConfigSetProcessor-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 3239021 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 3239022 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56306_yw%2Fvq
   [junit4]   2> 3239023 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:56306_yw%2Fvq
   [junit4]   2> 3239026 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 3239026 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3239026 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 3239060 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 3239060 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 3239062 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores
   [junit4]   2> 3239063 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 3239063 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1
   [junit4]   2> 3239063 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 3239064 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 3239064 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 3239064 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 3239064 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 3239066 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 3239067 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:56306/yw/vq",
   [junit4]   2>   "node_name":"127.0.0.1:56306_yw%2Fvq",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 3239067 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:56306/yw/vq",
   [junit4]   2>   "node_name":"127.0.0.1:56306_yw%2Fvq",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 3239067 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 3239068 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 3239069 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3239069 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 3239071 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3240072 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3240073 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 3240073 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 3240073 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 3240075 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 3240075 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1'
   [junit4]   2> 3240075 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3240075 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 3240077 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 3240082 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 3240085 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 3240093 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3240113 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3240196 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3240201 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 3240202 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 3240215 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 3240217 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 3240220 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3240222 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3240224 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0
   [junit4]   2> 3240225 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 3240225 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 3240225 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1/data/]
   [junit4]   2> 3240225 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@63c817d6
   [junit4]   2> 3240225 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1/data
   [junit4]   2> 3240225 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1/data/index/
   [junit4]   2> 3240226 WARN  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 3240226 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1/data/index
   [junit4]   2> 3240226 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=61.474609375, floorSegmentMB=0.9794921875, forceMergeDeletesPctAllowed=12.689588335127715, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 3240227 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@16ea93ed lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6fd47c90; maxCacheMB=0.9072265625 maxMergeSizeMB=9.765625E-4)),segFN=segments_1,generation=1}
   [junit4]   2> 3240227 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 3240227 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1/data/
   [junit4]   2> 3240227 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 3240230 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3240230 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3240231 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3240231 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3240231 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3240231 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3240231 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3240231 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3240231 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3240231 WARN  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 3240233 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3240234 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3240235 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3240236 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3240239 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/dump,/update/csv,/sql,/graph,/admin/segments,/admin/system,/replication,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file
   [junit4]   2> 3240239 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 3240240 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3240240 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3240240 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3240240 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3240241 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=28, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 3240241 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@16ea93ed lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6fd47c90; maxCacheMB=0.9072265625 maxMergeSizeMB=9.765625E-4)),segFN=segments_1,generation=1}
   [junit4]   2> 3240241 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 3240241 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1ca6a65d[collection1] main]
   [junit4]   2> 3240241 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 3240242 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 3240242 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3240243 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3240243 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 3240243 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3240244 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3240244 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3240244 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 3240244 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3240244 INFO  (searcherExecutor-7992-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1ca6a65d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3240245 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 3240245 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 3240245 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 3240245 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1536087412237664256
   [junit4]   2> 3240247 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1536087412237664256
   [junit4]   2> 3240247 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 3240248 INFO  (coreLoadExecutor-7991-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 3240248 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:56306/yw/vq collection:control_collection shard:shard1
   [junit4]   2> 3240250 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 3240255 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/96005463154032644-core_node1-n_0000000000
   [junit4]   2> 3240256 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 3240260 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3240260 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3240261 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:56306/yw/vq/collection1/
   [junit4]   2> 3240261 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3240261 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:56306/yw/vq/collection1/ has no replicas
   [junit4]   2> 3240261 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 3240263 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 3240268 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/96005463154032644-core_node1-n_0000000000
   [junit4]   2> 3240271 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:56306/yw/vq/collection1/ shard1
   [junit4]   2> 3240272 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:56306/yw/vq",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 3240376 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3240423 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:56306/yw/vq/collection1/ and leader is http://127.0.0.1:56306/yw/vq/collection1/
   [junit4]   2> 3240424 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56306/yw/vq
   [junit4]   2> 3240424 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3240424 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 3240424 INFO  (coreZkRegister-7985-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56306_yw%2Fvq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 3240427 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:56306/yw/vq",
   [junit4]   2>   "node_name":"127.0.0.1:56306_yw%2Fvq",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 3240427 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:56306/yw/vq",
   [junit4]   2>   "node_name":"127.0.0.1:56306_yw%2Fvq",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 3240535 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3240572 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:65284/solr
   [junit4]   2> 3240573 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3240573 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3240574 INFO  (zkCallback-21503-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4a100dc2 name:ZooKeeperConnection Watcher:127.0.0.1:65284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3240575 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3240575 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3240575 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3240576 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 3240577 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3240579 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3240951 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001/cores/collection1
   [junit4]   2> 3240952 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001
   [junit4]   2> 3240952 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 3240954 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1a91f33c{/yw/vq,null,AVAILABLE}
   [junit4]   2> 3240955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.ServerConnector Started ServerConnector@77383096{HTTP/1.1,[http/1.1]}{127.0.0.1:38882}
   [junit4]   2> 3240955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.Server Started @3245891ms
   [junit4]   2> 3240955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/yw/vq, hostPort=38882, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001/cores}
   [junit4]   2> 3240955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 3240955 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001'
   [junit4]   2> 3240956 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3240956 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 3240956 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3240957 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3240959 INFO  (zkCallback-21504-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@12cbbc43 name:ZooKeeperConnection Watcher:127.0.0.1:65284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3240959 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3240960 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3240962 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3240962 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001/solr.xml
   [junit4]   2> 3240967 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001/cores
   [junit4]   2> 3240967 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.CoreContainer New CoreContainer 2094567899
   [junit4]   2> 3240967 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001]
   [junit4]   2> 3240967 WARN  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001/lib
   [junit4]   2> 3240967 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 3240969 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 3240969 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3240969 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3240970 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65284/solr
   [junit4]   2> 3240970 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3240970 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3240971 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3240972 INFO  (zkCallback-21507-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@25f8a1d name:ZooKeeperConnection Watcher:127.0.0.1:65284 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3240973 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3240973 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3240974 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$50/1084092672@40340443
   [junit4]   2> 3240975 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3240976 INFO  (zkCallback-21508-thread-1-processing-n:127.0.0.1:38882_yw%2Fvq) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@457144fe name:ZooKeeperConnection Watcher:127.0.0.1:65284/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3240976 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3240985 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3240986 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 3240987 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3240992 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3240993 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96005463154032648-127.0.0.1:38882_yw%2Fvq-n_0000000001
   [junit4]   2> 3240994 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000 to know if I could be the leader
   [junit4]   2> 3240994 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.ZkController Publish node=127.0.0.1:38882_yw%2Fvq as DOWN
   [junit4]   2> 3240997 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:38882_yw%2Fvq"} current state version: 3
   [junit4]   2> 3240997 INFO  (OverseerStateUpdate-96005463154032644-127.0.0.1:56306_yw%2Fvq-n_0000000000) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:38882_yw%2Fvq
   [junit4]   2> 3241109 INFO  (zkCallback-21503-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3241109 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3241109 INFO  (zkCallback-21508-thread-1-processing-n:127.0.0.1:38882_yw%2Fvq) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3241997 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38882_yw%2Fvq
   [junit4]   2> 3241998 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:38882_yw%2Fvq
   [junit4]   2> 3242000 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3242000 INFO  (zkCallback-21503-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3242000 INFO  (zkCallback-21508-thread-1-processing-n:127.0.0.1:38882_yw%2Fvq) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3242001 INFO  (zkCallback-21499-thread-1-processing-n:127.0.0.1:56306_yw%2Fvq) [n:127.0.0.1:56306_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3242001 INFO  (zkCallback-21508-thread-1-processing-n:127.0.0.1:38882_yw%2Fvq) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3242001 INFO  (zkCallback-21503-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3242017 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 3242017 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 3242018 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001/shard-1-001/cores
   [junit4]   2> 3242020 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [n:127.0.0.1:38882_yw%2Fvq    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadO

[...truncated too long message...]

.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 3246909 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@77383096{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 3246909 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1a91f33c{/yw/vq,null,UNAVAILABLE}
   [junit4]   2> 3246910 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 52603
   [junit4]   2> 3246910 INFO  (TEST-TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue-seed#[CFC46D10AFC7D504]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:65284 65284
   [junit4]   2> 3246944 INFO  (Thread-8368) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:65284 65284
   [junit4]   2> 3246957 WARN  (Thread-8368) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/aliases.json
   [junit4]   2> 	4	/solr/clusterprops.json
   [junit4]   2> 	3	/solr/security.json
   [junit4]   2> 	3	/solr/configs/conf1
   [junit4]   2> 	2	/solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/live_nodes
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 	2	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDownShardTolerantSearch -Dtests.method=searchingShouldFailWithoutTolerantSearchSetToTrue -Dtests.seed=CFC46D10AFC7D504 -Dtests.slow=true -Dtests.locale=en-PH -Dtests.timezone=Eire -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 8.42s J0 | TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Error message from server should have the name of the down shard
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CFC46D10AFC7D504:FD14CC1BAA65B465]:0)
   [junit4]    > 	at org.apache.solr.cloud.TestDownShardTolerantSearch.searchingShouldFailWithoutTolerantSearchSetToTrue(TestDownShardTolerantSearch.java:68)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3246974 INFO  (SUITE-TestDownShardTolerantSearch-seed#[CFC46D10AFC7D504]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestDownShardTolerantSearch_CFC46D10AFC7D504-001
   [junit4]   2> Jun 03, 2016 4:11:59 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=10, maxDocsPerChunk=1, blockSize=862), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=10, blockSize=862)), sim=ClassicSimilarity, locale=en-PH, timezone=Eire
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=3,threads=1,free=271141624,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestReplicationHandler, JsonLoaderTest, SimplePostToolTest, TestSolrDeletionPolicy1, SliceStateTest, StressHdfsTest, HdfsRestartWhileUpdatingTest, CollectionsAPIAsyncDistributedZkTest, TestXmlQParser, OutOfBoxZkACLAndCredentialsProvidersTest, DisMaxRequestHandlerTest, SpatialHeatmapFacetsTest, SaslZkACLProviderTest, SchemaVersionSpecificBehaviorTest, StandardRequestHandlerTest, DOMUtilTest, TestMiniSolrCloudClusterSSL, TestFieldCacheSanityChecker, TestArbitraryIndexDir, TestExtendedDismaxParser, MoreLikeThisHandlerTest, DocumentAnalysisRequestHandlerTest, TestCollationFieldDocValues, TestSweetSpotSimilarityFactory, StatelessScriptUpdateProcessorFactoryTest, AnalyticsMergeStrategyTest, TestDocSet, TestLFUCache, RecoveryZkTest, SampleTest, SimpleMLTQParserTest, TestMergePolicyConfig, TestJsonRequest, DirectSolrSpellCheckerTest, WrapperMergePolicyFactoryTest, RankQueryTest, SuggesterFSTTest, HdfsRecoverLeaseTest, TestRecoveryHdfs, DistributedFacetPivotLongTailTest, TestSchemaSimilarityResource, IndexBasedSpellCheckerTest, SOLR749Test, SystemInfoHandlerTest, TestBulkSchemaConcurrent, ReturnFieldsTest, TestFieldCacheWithThreads, TestConfig, TestJmxMonitoredMap, TestMaxScoreQueryParser, TestConfigSetsAPIExclusivity, ZkControllerTest, TestCloudBackupRestore, TestElisionMultitermQuery, XmlUpdateRequestHandlerTest, ZkCLITest, SimpleCollectionCreateDeleteTest, DeleteLastCustomShardedReplicaTest, CursorPagingTest, TestScoreJoinQPNoScore, HdfsChaosMonkeyNothingIsSafeTest, HdfsDirectoryFactoryTest, SolrRequestParserTest, DirectUpdateHandlerTest, OverseerTest, XsltUpdateRequestHandlerTest, TestCSVLoader, BadComponentTest, TestQueryWrapperFilter, OverseerStatusTest, TestSubQueryTransformerDistrib, TestManagedStopFilterFactory, TestBlendedInfixSuggestions, HighlighterMaxOffsetTest, RequiredFieldsTest, ConcurrentDeleteAndCreateCollectionTest, UUIDFieldTest, TestSearchPerf, DistributedQueryElevationComponentTest, CustomCollectionTest, TestPseudoReturnFields, BlockJoinFacetDistribTest, EnumFieldTest, TestSortingResponseWriter, HardAutoCommitTest, ClusterStateTest, CdcrRequestHandlerTest, TestCopyFieldCollectionResource, PathHierarchyTokenizerFactoryTest, HdfsLockFactoryTest, TestCoreContainer, DocValuesTest, DistributedSpellCheckComponentTest, HdfsUnloadDistributedZkTest, CircularListTest, TestEmbeddedSolrServerConstructors, TestTrackingShardHandlerFactory, HttpPartitionTest, TestStressLiveNodes, BlockCacheTest, BlockJoinFacetSimpleTest, MBeansHandlerTest, ZkStateReaderTest, BigEndianAscendingWordSerializerTest, TestLMDirichletSimilarityFactory, TestUninvertingReader, URLClassifyProcessorTest, SparseHLLTest, TestFunctionQuery, TestReplicaProperties, TestStressVersions, SolrIndexConfigTest, QueryElevationComponentTest, TestExactSharedStatsCache, DistributedExpandComponentTest, CloudExitableDirectoryReaderTest, SolrTestCaseJ4Test, TestNumericTerms32, TestDynamicFieldCollectionResource, TestManagedResourceStorage, DistanceUnitsTest, TestSerializedLuceneMatchVersion, TestStressReorder, DistributedQueryComponentCustomSortTest, HdfsCollectionsAPIDistributedZkTest, TestExpandComponent, TestShardHandlerFactory, JSONWriterTest, DateRangeFieldTest, TestInfoStreamLogging, TimeZoneUtilsTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, RegexBoostProcessorTest, TestShortCircuitedRequests, ReplicationFactorTest, ScriptEngineTest, CurrencyFieldXmlFileTest, BufferStoreTest, TestTestInjection, RulesTest, PolyFieldTest, TestConfigSetProperties, TestFieldTypeCollectionResource, ExplicitHLLTest, CollectionStateFormat2Test, DeleteReplicaTest, DeleteStatusTest, DistribJoinFromCollectionTest, DistributedVersionInfoTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnCommitTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, RecoveryAfterSoftCommitTest, RollingRestartTest, SSLMigrationTest, SharedFSAutoReplicaFailoverTest, SolrCloudExampleTest, TestCloudInspectUtil, TestCloudPivotFacet, TestCollectionAPI, TestConfigSetsAPI, TestCryptoKeys, TestDownShardTolerantSearch]
   [junit4] Completed [474/612 (1!)] on J0 in 8.96s, 1 test, 1 failure <<< FAILURES!

[...truncated 439 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:740: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:531: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1427: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2560 tests, 1 failure, 88 ignored (73 assumptions) [seed: CFC46D10AFC7D504]

Total time: 85 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 629 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/629/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

28 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.test

Error Message:
Error from server at http://127.0.0.1:45639: Error CREATEing SolrCore 'onenodecollectioncore': Unable to create core [onenodecollectioncore] Caused by: Timed out getting coreNodeName for onenodecollectioncore

Stack Trace:
java.lang.AssertionError: Error from server at http://127.0.0.1:45639: Error CREATEing SolrCore 'onenodecollectioncore': Unable to create core [onenodecollectioncore] Caused by: Timed out getting coreNodeName for onenodecollectioncore
	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:F700CFE632204E23]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:162)
	at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:51302/ff_/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /ff_/collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51302/ff_/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /ff_/collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:F700CFE632204E23]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Error from server at http://127.0.0.1:59606/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:59606/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:F700CFE632204E23]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
Error from server at http://127.0.0.1:51217/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51217/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:F700CFE632204E23]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tryDelete(ChaosMonkeySafeLeaderTest.java:190)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration

Error Message:
Error CREATEing SolrCore 'testcore': Unable to create core [testcore] Caused by: Timed out getting shard id for core: testcore

Stack Trace:
org.apache.solr.common.SolrException: Error CREATEing SolrCore 'testcore': Unable to create core [testcore] Caused by: Timed out getting shard id for core: testcore
	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:C1DF9693E5A62DEE]:0)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:808)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:741)
	at org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration(ClusterStateUpdateTest.java:158)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to create core [testcore]
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:851)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:774)
	... 41 more
Caused by: org.apache.solr.common.SolrException: Timed out getting shard id for core: testcore
	at org.apache.solr.cloud.ZkController.waitForShardId(ZkController.java:1407)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1362)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	... 42 more


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:45145/fsb/rs

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:45145/fsb/rs
	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:F700CFE632204E23]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:617)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testNoConfigSetExist(CollectionsAPIDistributedZkTest.java:602)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:177)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
	... 47 more


FAILED:  org.apache.solr.cloud.DistribCursorPagingTest.test

Error Message:
Error from server at http://127.0.0.1:45222/tn_uur/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /tn_uur/collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45222/tn_uur/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /tn_uur/collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:F700CFE632204E23]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.indexDoc(AbstractFullDistribZkTestBase.java:775)
	at org.apache.solr.cloud.DistribCursorPagingTest.doBadInputTest(DistribCursorPagingTest.java:111)
	at org.apache.solr.cloud.DistribCursorPagingTest.test(DistribCursorPagingTest.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Error from server at http://127.0.0.1:44369/k_/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /k_/collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:44369/k_/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /k_/collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:F700CFE632204E23]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.indexDoc(AbstractFullDistribZkTestBase.java:775)
	at org.apache.solr.BaseDistributedSearchTestCase.indexr(BaseDistributedSearchTestCase.java:465)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.addUpdateDelete(FullSolrCloudDistribCmdsTest.java:472)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:81)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout

Error Message:
Could not start container:7000. Reason:class org.apache.solr.common.SolrException Error CREATEing SolrCore 'collection1_shard1': Unable to create core [collection1_shard1] Caused by: Timed out getting coreNodeName for collection1_shard1

Stack Trace:
java.lang.AssertionError: Could not start container:7000. Reason:class org.apache.solr.common.SolrException Error CREATEing SolrCore 'collection1_shard1': Unable to create core [collection1_shard1] Caused by: Timed out getting coreNodeName for collection1_shard1
	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:D663559FBBF9AB30]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.LeaderElectionIntegrationTest.setUp(LeaderElectionIntegrationTest.java:102)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:905)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderElectionIntegrationTest.testSimpleSliceLeaderElection

Error Message:
Could not start container:7000. Reason:class org.apache.solr.common.SolrException Error CREATEing SolrCore 'collection1_shard1': Unable to create core [collection1_shard1] Caused by: Timed out getting coreNodeName for collection1_shard1

Stack Trace:
java.lang.AssertionError: Could not start container:7000. Reason:class org.apache.solr.common.SolrException Error CREATEing SolrCore 'collection1_shard1': Unable to create core [collection1_shard1] Caused by: Timed out getting coreNodeName for collection1_shard1
	at __randomizedtesting.SeedInfo.seed([7F54F03C9CDC23DB:21603D72DFA3BF3C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.LeaderElectionIntegrationTest.setUp(LeaderElectionIntegrationTest.java:102)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:905)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.Randomized

[...truncated too long message...]

emp/solr.cloud.ShardSplitTest_7F54F03C9CDC23DB-001/shard-4-002/cores/collection1/data [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ShardSplitTest_7F54F03C9CDC23DB-001/shard-4-002/cores/collection1/data;done=false>>]
   [junit4]   2> 18823087 INFO  (coreCloseExecutor-5619-thread-1) [n:127.0.0.1:56144_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 18823087 INFO  (coreCloseExecutor-5619-thread-1) [n:127.0.0.1:56144_ c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ShardSplitTest_7F54F03C9CDC23DB-001/shard-4-002/cores/collection1/data
   [junit4]   2> 18823087 INFO  (OverseerStateUpdate-96020752313352209-127.0.0.1:56144_-n_0000000004) [n:127.0.0.1:56144_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 18823088 INFO  (TEST-ShardSplitTest.testSplitShardWithRule-seed#[7F54F03C9CDC23DB]) [    ] o.a.s.c.Overseer Overseer (id=96020752313352209-127.0.0.1:56144_-n_0000000004) closing
   [junit4]   2> 18823088 INFO  (OverseerStateUpdate-96020752313352209-127.0.0.1:56144_-n_0000000004) [n:127.0.0.1:56144_    ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:56144_
   [junit4]   2> 18823091 INFO  (zkCallback-2623-thread-3) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 18823092 INFO  (zkCallback-2623-thread-3) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
   [junit4]   2> 18823092 INFO  (zkCallback-2647-thread-2-processing-n:127.0.0.1:56144_) [n:127.0.0.1:56144_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 18823092 WARN  (zkCallback-2647-thread-2-processing-n:127.0.0.1:56144_) [n:127.0.0.1:56144_    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 18823098 INFO  (TEST-ShardSplitTest.testSplitShardWithRule-seed#[7F54F03C9CDC23DB]) [    ] o.e.j.s.ServerConnector Stopped ServerConnector@49b69034{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 18823098 INFO  (TEST-ShardSplitTest.testSplitShardWithRule-seed#[7F54F03C9CDC23DB]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@40f3bb0{/,null,UNAVAILABLE}
   [junit4]   2> 18823101 INFO  (TEST-ShardSplitTest.testSplitShardWithRule-seed#[7F54F03C9CDC23DB]) [    ] o.a.s.c.ChaosMonkey monkey: stop shard! 47278
   [junit4]   2> 18823103 INFO  (TEST-ShardSplitTest.testSplitShardWithRule-seed#[7F54F03C9CDC23DB]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33417 33417
   [junit4]   2> 18823116 INFO  (Thread-5267) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33417 33417
   [junit4]   2> 18823120 WARN  (Thread-5267) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	6	/solr/clusterprops.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	5	/solr/configs/conf1
   [junit4]   2> 	4	/solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterprops.json
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/collections/shardSplitWithRule/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/live_nodes
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 	5	/solr/overseer/collection-queue-work
   [junit4]   2> 	4	/solr/overseer/queue
   [junit4]   2> 	3	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> 18823121 INFO  (SUITE-ShardSplitTest-seed#[7F54F03C9CDC23DB]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.ShardSplitTest_7F54F03C9CDC23DB-001
   [junit4]   2> Jun 05, 2016 9:10:46 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 4 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=1497, maxMBSortInHeap=6.935577567028022, sim=ClassicSimilarity, locale=zh, timezone=America/Anchorage
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=3,threads=1,free=178591208,total=433594368
   [junit4]   2> NOTE: All tests run in this JVM: [SuggesterWFSTTest, TestAuthorizationFramework, DistributedDebugComponentTest, DeleteReplicaTest, TestLegacyFieldCache, TestGeoJSONResponseWriter, HdfsBasicDistributedZkTest, TestScoreJoinQPScore, BlockDirectoryTest, CacheHeaderTest, TestJoin, TestSolrConfigHandlerCloud, CurrencyFieldOpenExchangeTest, OverseerRolesTest, SynonymTokenizerTest, TestCharFilters, TestConfigSetsAPIZkFailure, CdcrVersionReplicationTest, TestRequestForwarding, SolrCloudExampleTest, DateMathParserTest, TestQueryUtils, FacetPivotSmallTest, AddBlockUpdateTest, DistributedQueueTest, TestFieldSortValues, CloudMLTQParserTest, TestTrieFacet, TestSolrXml, CachingDirectoryFactoryTest, TestSlowCompositeReaderWrapper, TestNamedUpdateProcessors, TestRangeQuery, HLLUtilTest, DebugComponentTest, TestJsonFacets, MinimalSchemaTest, ExitableDirectoryReaderTest, TestMissingGroups, UUIDUpdateProcessorFallbackTest, HdfsNNFailoverTest, DocValuesMissingTest, TestRandomCollapseQParserPlugin, TestSolrQueryParser, DefaultValueUpdateProcessorTest, TestRestManager, TestFieldTypeResource, ResourceLoaderTest, BaseCdcrDistributedZkTest, RAMDirectoryFactoryTest, TestConfigSetsAPI, TestAnalyzeInfixSuggestions, TestLMJelinekMercerSimilarityFactory, JavabinLoaderTest, DeleteInactiveReplicaTest, TestHighFrequencyDictionaryFactory, DeleteStatusTest, GraphQueryTest, ConfigSetsAPITest, TestUniqueKeyFieldResource, TestManagedSynonymFilterFactory, TestReRankQParserPlugin, ZkNodePropsTest, AnalyticsQueryTest, DistributedTermsComponentTest, TestFieldCacheSort, ShardRoutingCustomTest, ChaosMonkeySafeLeaderTest, TestDefaultSearchFieldResource, TestZkChroot, TestAuthenticationFramework, TestObjectReleaseTracker, TestUtils, BadIndexSchemaTest, TestSSLRandomization, CheckHdfsIndexTest, DistributedIntervalFacetingTest, ResponseLogComponentTest, OutputWriterTest, TestRebalanceLeaders, SpellCheckComponentTest, BinaryUpdateRequestHandlerTest, FileUtilsTest, TestExclusionRuleCollectionAccess, TestCloudManagedSchema, BasicFunctionalityTest, ConvertedLegacyTest, ZkStateWriterTest, TestDistribDocBasedVersion, ForceLeaderTest, TestSolrCoreProperties, UniqFieldsUpdateProcessorFactoryTest, TestLeaderInitiatedRecoveryThread, TestSolrCLIRunExample, TestWriterPerf, SmileWriterTest, CollectionTooManyReplicasTest, CoreAdminCreateDiscoverTest, SolrCoreCheckLockOnStartupTest, UpdateRequestProcessorFactoryTest, ResponseHeaderTest, CdcrUpdateLogTest, TestInitParams, TestFastOutputStream, DistributedFacetPivotWhiteBoxTest, TestPHPSerializedResponseWriter, ClassificationUpdateProcessorFactoryTest, TestDocBasedVersionConstraints, BlobRepositoryCloudTest, TestSolrQueryParserDefaultOperatorResource, TestOmitPositions, TestSchemaResource, TestReloadDeadlock, MultiThreadedOCPTest, BJQParserTest, TestOverriddenPrefixQueryForCustomFieldType, TestManagedResource, WordBreakSolrSpellCheckerTest, TestFieldCacheReopen, BasicDistributedZk2Test, TestClassicSimilarityFactory, SolrInfoMBeanTest, TestSolrIndexConfig, StatsComponentTest, TestSchemaVersionResource, DistributedMLTComponentTest, DocExpirationUpdateProcessorFactoryTest, CleanupOldIndexTest, TestFiltering, BasicAuthIntegrationTest, TestStandardQParsers, TestCloudDeleteByQuery, TestRawResponseWriter, TestFileDictionaryLookup, TestExactStatsCache, FullSolrCloudDistribCmdsTest, TermVectorComponentTest, TestManagedSchema, TestSolrDeletionPolicy2, TestSolr4Spatial, HdfsRecoveryZkTest, FullHLLTest, SpellCheckCollatorTest, ShardSplitTest]
   [junit4] Completed [607/612 (22!)] on J1 in 1257.91s, 2 tests, 1 error <<< FAILURES!

[...truncated 34 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:740: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:531: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1427: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2560 tests, 2 suite-level errors, 21 errors, 5 failures, 1218 ignored (70 assumptions) [seed: 7F54F03C9CDC23DB]

Total time: 332 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 628 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/628/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

17 tests failed.
FAILED:  org.apache.solr.cloud.AliasIntegrationTest.test

Error Message:
Error from server at http://127.0.0.1:35962/_l/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /_l/collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:35962/_l/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /_l/collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:99F053018D82AEB3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.AliasIntegrationTest.test(AliasIntegrationTest.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.test

Error Message:
Error from server at http://127.0.0.1:42310/wz_ce/z: Error CREATEing SolrCore 'onenodecollectioncore': Unable to create core [onenodecollectioncore] Caused by: Timed out getting coreNodeName for onenodecollectioncore

Stack Trace:
java.lang.AssertionError: Error from server at http://127.0.0.1:42310/wz_ce/z: Error CREATEing SolrCore 'onenodecollectioncore': Unable to create core [onenodecollectioncore] Caused by: Timed out getting coreNodeName for onenodecollectioncore
	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:99F053018D82AEB3]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:162)
	at org.apache.solr.cloud.BasicDistributedZk2Test.test(BasicDistributedZk2Test.java:71)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:49617/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:49617/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:99F053018D82AEB3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
Error from server at http://127.0.0.1:62305/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:62305/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:99F053018D82AEB3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
Error from server at http://127.0.0.1:48529/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:48529/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:99F053018D82AEB3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:895)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:858)
	at org.apache.solr.client.solrj.SolrClient.deleteByQuery(SolrClient.java:873)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.del(AbstractFullDistribZkTestBase.java:835)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.tryDelete(ChaosMonkeySafeLeaderTest.java:190)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:92)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:43361/_j/c

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:43361/_j/c
	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:99F053018D82AEB3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:617)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:400)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testNoConfigSetExist(CollectionsAPIDistributedZkTest.java:602)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:177)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:511)
	... 47 more


FAILED:  org.apache.solr.cloud.DistribCursorPagingTest.test

Error Message:
Error from server at http://127.0.0.1:61490/ph_jfl/s/collection1: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /ph_jfl/s/collection1/update. Reason: <pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1  at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)  at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)  at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)  at org.eclipse.jetty.server.Server.handle(Server.java:518)  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)  at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)  at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)  at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)  at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1  at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)  at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)  at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)  at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  ... 1 more ,code=500}</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/> </body> </html> 

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:61490/ph_jfl/s/collection1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 500 </title>
</head>
<body>
<h2>HTTP ERROR: 500</h2>
<p>Problem accessing /ph_jfl/s/collection1/update. Reason:
<pre>    {metadata={error-class=org.apache.solr.common.SolrException,root-error-class=org.apache.solr.common.SolrException},msg=SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1,trace=org.apache.solr.common.SolrException: SolrCore &apos;collection1&apos; is not available due to init failure: Timed out getting coreNodeName for collection1
	at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1103)
	at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:257)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:419)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:111)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	... 1 more
,code=500}</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
</body>
</html>

	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:99F053018D82AEB3]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.indexDoc(AbstractFullDistribZkTestBase.java:775)
	at org.apache.solr.cloud.DistribCursorPagingTest.doBadInputTest(DistribCursorPagingTest.java:111)
	at org.apache.solr.cloud.DistribCursorPagingTest.test(DistribCursorPagingTest.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout

Error Message:
Could not start container:7000. Reason:class org.apache.solr.common.SolrException Error CREATEing SolrCore 'collection1_shard1': Unable to create core [collection1_shard1] Caused by: Timed out getting coreNodeName for collection1_shard1

Stack Trace:
java.lang.AssertionError: Could not start container:7000. Reason:class org.apache.solr.common.SolrException Error CREATEing SolrCore 'collection1_shard1': Unable to create core [collection1_shard1] Caused by: Timed out getting coreNodeName for collection1_shard1
	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:B893C978045B4BA0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.LeaderElectionIntegrationTest.setUp(LeaderElectionIntegrationTest.java:102)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:905)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderElectionIntegrationTest.testSimpleSliceLeaderElection

Error Message:
Could not start container:7000. Reason:class org.apache.solr.common.SolrException Error CREATEing SolrCore 'collection1_shard1': Unable to create core [collection1_shard1] Caused by: Timed out getting coreNodeName for collection1_shard1

Stack Trace:
java.lang.AssertionError: Could not start container:7000. Reason:class org.apache.solr.common.SolrException Error CREATEing SolrCore 'collection1_shard1': Unable to create core [collection1_shard1] Caused by: Timed out getting coreNodeName for collection1_shard1
	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:4F90A19560015FAC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.LeaderElectionIntegrationTest.setUp(LeaderElectionIntegrationTest.java:102)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:905)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionIntegrationTest

Error Message:
21 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionIntegrationTest:     1) Thread[id=10668, name=OverseerCollectionConfigSetProcessor-96015671684300803-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:328)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:167)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=10632, name=Connection evictor, state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=10636, name=TEST-LeaderElectionIntegrationTest.testLeaderElectionAfterClientTimeout-seed#[11A46CDB237EC34B]-SendThread(127.0.0.1:52418), state=TIMED_WAITING, group=TGRP-LeaderElectionIntegrationTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    4) Thread[id=10669, name=OverseerHdfsCoreFailoverThread-96015671684300803-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=10667, name=OverseerStateUpdate-96015671684300803-127.0.0.1:7000_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:102)         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)         at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)         at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:309)         at org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:288)         at org.apache.solr.cloud.DistributedQueue.firstElement(DistributedQueue.java:357)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:164)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:142)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overse

[...truncated too long message...]

   > 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]    > 	at org.eclipse.jetty.server.Server.handle(Server.java:518)
   [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]    > 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
   [junit4]    > 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]    > 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]    > 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]    > 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
   [junit4]    > 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.common.SolrException: Timed out getting coreNodeName for collection1
   [junit4]    > 	at org.apache.solr.cloud.ZkController.waitForCoreNodeName(ZkController.java:1387)
   [junit4]    > 	at org.apache.solr.cloud.ZkController.doGetShardIdAndNodeNameProcess(ZkController.java:1365)
   [junit4]    > 	at org.apache.solr.cloud.ZkController.preRegister(ZkController.java:1456)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:832)
   [junit4]    > 	at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:491)
   [junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	... 1 more
   [junit4]    > ,code=500}</pre></p>
   [junit4]    > <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.8.v20160314</a><hr/>
   [junit4]    > </body>
   [junit4]    > </html>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A46CDB237EC34B:99F053018D82AEB3]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:574)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.indexDoc(AbstractFullDistribZkTestBase.java:775)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.index(BaseDistributedSearchTestCase.java:476)
   [junit4]    > 	at org.apache.solr.cloud.ShardRoutingTest.doAddDoc(ShardRoutingTest.java:324)
   [junit4]    > 	at org.apache.solr.cloud.ShardRoutingTest.doHashingTest(ShardRoutingTest.java:132)
   [junit4]    > 	at org.apache.solr.cloud.ShardRoutingTest.test(ShardRoutingTest.java:109)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 11508545 INFO  (SUITE-ShardRoutingTest-seed#[11A46CDB237EC34B]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.ShardRoutingTest_11A46CDB237EC34B-001
   [junit4]   2> Jun 04, 2016 11:39:40 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=501, maxMBSortInHeap=5.091077782666301, sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=es-VE, timezone=Europe/Andorra
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=3,threads=1,free=198483888,total=488636416
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldCache, ExplicitHLLTest, TestXIncludeConfig, HdfsDirectoryFactoryTest, SchemaVersionSpecificBehaviorTest, DocValuesTest, SolrRequestParserTest, TestTolerantSearch, BlockCacheTest, SharedFSAutoReplicaFailoverTest, ConnectionManagerTest, TestBlendedInfixSuggestions, StandardRequestHandlerTest, TestExtendedDismaxParser, BasicDistributedZkTest, DocValuesMultiTest, TestCustomDocTransformer, DataDrivenBlockJoinTest, TestMaxScoreQueryParser, PrimitiveFieldTypeTest, AssignTest, TestSolrDynamicMBean, TestUninvertingReader, RequestLoggingTest, DirectSolrSpellCheckerTest, TestStressVersions, TestElisionMultitermQuery, TestReqParamsAPI, TestComponentsName, SegmentsInfoRequestHandlerTest, QueryEqualityTest, TestSolrFieldCacheMBean, HdfsCollectionsAPIDistributedZkTest, ZkCLITest, CollectionsAPIAsyncDistributedZkTest, BufferStoreTest, SpatialFilterTest, TestPKIAuthenticationPlugin, UUIDFieldTest, ActionThrottleTest, OverseerCollectionConfigSetProcessorTest, TestFunctionQuery, TestBulkSchemaConcurrent, TestCollapseQParserPlugin, CollectionStateFormat2Test, DateRangeFieldTest, AnalyticsMergeStrategyTest, DistributedSuggestComponentTest, TestGroupingSearch, DOMUtilTest, TestCSVLoader, UpdateParamsTest, CdcrRequestHandlerTest, ReturnFieldsTest, TestSortByMinMaxFunction, TestJsonRequest, TestNoOpRegenerator, TestMiniSolrCloudClusterBase, TriLevelCompositeIdRoutingTest, TestMinMaxOnMultiValuedField, HighlighterTest, TestStressLiveNodes, TestFreeTextSuggestions, TestSearchPerf, TestSubQueryTransformer, TestDFRSimilarityFactory, TestConfig, DistribCursorPagingTest, TestDynamicLoading, CoreAdminRequestStatusTest, SolrCoreTest, TestConfigSetsAPIExclusivity, TestPseudoReturnFields, TestAddFieldRealTimeGet, CloneFieldUpdateProcessorFactoryTest, TestPerFieldSimilarityWithDefaultOverride, DisMaxRequestHandlerTest, DistributedQueryComponentCustomSortTest, ReplicationFactorTest, TestCloudBackupRestore, DocumentBuilderTest, SpatialHeatmapFacetsTest, TestConfigReload, TestLockTree, DistributedFacetPivotSmallTest, DistributedVersionInfoTest, ChaosMonkeyNothingIsSafeTest, TestSurroundQueryParser, TestFaceting, BigEndianAscendingWordSerializerTest, TestSweetSpotSimilarityFactory, TestFilteredDocIdSet, FastVectorHighlighterTest, BadCopyFieldTest, TestMiniSolrCloudClusterSSL, XsltUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestUseDocValuesAsStored, TestLeaderElectionZkExpiry, TestSolrConfigHandlerConcurrent, TestCollectionAPI, TestStressLucene, ExternalFileFieldSortTest, AnalysisErrorHandlingTest, JSONWriterTest, TestIntervalFaceting, HighlighterMaxOffsetTest, BlockJoinFacetSimpleTest, DistributedExpandComponentTest, DeleteLastCustomShardedReplicaTest, SaslZkACLProviderTest, TestReplicaProperties, PKIAuthenticationIntegrationTest, CloudExitableDirectoryReaderTest, TestMergePolicyConfig, DocumentAnalysisRequestHandlerTest, TestCollationFieldDocValues, TestDocumentBuilder, RecoveryAfterSoftCommitTest, DistributedSpellCheckComponentTest, TestReplicationHandler, ZkStateReaderTest, ConjunctionSolrSpellCheckerTest, TestExactSharedStatsCache, SOLR749Test, TestFieldTypeCollectionResource, TestDynamicFieldCollectionResource, TestTolerantUpdateProcessorRandomCloud, SoftAutoCommitTest, TestSolrCloudWithKerberosAlt, HdfsBasicDistributedZk2Test, TestSort, TestRawTransformer, TestSerializedLuceneMatchVersion, DistributedFacetPivotLongTailTest, TestJmxIntegration, TestFieldCacheSanityChecker, DistribJoinFromCollectionTest, TestDocSet, TestDownShardTolerantSearch, TestFastLRUCache, TestTrackingShardHandlerFactory, VersionInfoTest, URLClassifyProcessorTest, TestLMDirichletSimilarityFactory, HdfsUnloadDistributedZkTest, TestSchemaNameResource, BadComponentTest, HdfsThreadLeakTest, SyncSliceTest, ShardRoutingTest]
   [junit4] Completed [612/612 (14!)] on J0 in 621.37s, 1 test, 1 error <<< FAILURES!

[...truncated 19 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:740: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:531: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1427: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2560 tests, 2 suite-level errors, 11 errors, 4 failures, 1245 ignored (69 assumptions) [seed: 11A46CDB237EC34B]

Total time: 210 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 627 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/627/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

4 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup

Error Message:
Error from server at http://127.0.0.1:43599/gcm: create the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:43599/gcm: create the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([78DB6A8408AAFCFC:83BEE3E8CE555F7]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:606)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1596)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1553)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1608)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1549)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup(DeleteReplicaTest.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"core":"c8n_1x3_lf_shard1_replica3","base_url":"http://127.0.0.1:43502","node_name":"127.0.0.1:43502_","state":"active","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf)={   "replicationFactor":"3",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "core":"c8n_1x3_lf_shard1_replica2",           "base_url":"http://127.0.0.1:39081",           "node_name":"127.0.0.1:39081_",           "state":"down"},         "core_node2":{           "state":"down",           "base_url":"http://127.0.0.1:60584",           "core":"c8n_1x3_lf_shard1_replica1",           "node_name":"127.0.0.1:60584_"},         "core_node3":{           "core":"c8n_1x3_lf_shard1_replica3",           "base_url":"http://127.0.0.1:43502",           "node_name":"127.0.0.1:43502_",           "state":"active",           "leader":"true"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node3:{"core":"c8n_1x3_lf_shard1_replica3","base_url":"http://127.0.0.1:43502","node_name":"127.0.0.1:43502_","state":"active","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"c8n_1x3_lf_shard1_replica2",
          "base_url":"http://127.0.0.1:39081",
          "node_name":"127.0.0.1:39081_",
          "state":"down"},
        "core_node2":{
          "state":"down",
          "base_url":"http://127.0.0.1:60584",
          "core":"c8n_1x3_lf_shard1_replica1",
          "node_name":"127.0.0.1:60584_"},
        "core_node3":{
          "core":"c8n_1x3_lf_shard1_replica3",
          "base_url":"http://127.0.0.1:43502",
          "node_name":"127.0.0.1:43502_",
          "state":"active",
          "leader":"true"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
	at __randomizedtesting.SeedInfo.seed([78DB6A8408AAFCFC:F08F555EA6569104]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:170)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:57)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([78DB6A8408AAFCFC:8FA884DCCE42531A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1327)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy

Error Message:
expected:<2> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<1>
	at __randomizedtesting.SeedInfo.seed([78DB6A8408AAFCFC:D72044A5AC8145C0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:129)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy(ZkStateReaderTest.java:58)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10914 lines...]
   [junit4] Suite: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/init-core-data-001
   [junit4]   2> 782679 INFO  (SUITE-LeaderFailoverAfterPartitionTest-seed#[78DB6A8408AAFCFC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
   [junit4]   2> 782679 INFO  (SUITE-LeaderFailoverAfterPartitionTest-seed#[78DB6A8408AAFCFC]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 782699 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 782700 INFO  (Thread-2284) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 782700 INFO  (Thread-2284) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 782800 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.ZkTestServer start zk server on port:39279
   [junit4]   2> 782800 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 782801 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 782805 INFO  (zkCallback-514-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@50f65a74 name:ZooKeeperConnection Watcher:127.0.0.1:39279 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 782805 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 782805 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 782806 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 782812 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 782813 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 782815 INFO  (zkCallback-515-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5504248f name:ZooKeeperConnection Watcher:127.0.0.1:39279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 782815 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 782816 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 782816 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 782819 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 782822 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 782824 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 782827 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 782827 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 782831 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 782831 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 782834 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 782834 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 782837 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 782837 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 782840 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 782841 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 782859 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 782859 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 782861 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 782861 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 782863 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 782863 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 782865 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 782866 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 782868 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 782868 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 782870 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 782870 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 783215 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1
   [junit4]   2> 783219 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 783222 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@62470979{/,null,AVAILABLE}
   [junit4]   2> 783227 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.e.j.s.ServerConnector Started ServerConnector@5ca68b26{HTTP/1.1,[http/1.1]}{127.0.0.1:53966}
   [junit4]   2> 783228 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.e.j.s.Server Started @788062ms
   [junit4]   2> 783228 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/tempDir-001/control/data, hostContext=/, hostPort=43502, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores}
   [junit4]   2> 783228 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 783228 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001'
   [junit4]   2> 783229 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 783229 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 783229 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 783230 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 783234 INFO  (zkCallback-516-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7d497244 name:ZooKeeperConnection Watcher:127.0.0.1:39279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 783234 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 783235 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 783237 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 783237 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/solr.xml
   [junit4]   2> 783244 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores
   [junit4]   2> 783244 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.CoreContainer New CoreContainer 1218496486
   [junit4]   2> 783244 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001]
   [junit4]   2> 783244 WARN  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/lib
   [junit4]   2> 783244 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 783247 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 783248 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 783248 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 783248 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39279/solr
   [junit4]   2> 783249 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 783249 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 783249 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 783253 INFO  (zkCallback-519-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@bc3dba5 name:ZooKeeperConnection Watcher:127.0.0.1:39279 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 783256 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 783257 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 783260 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$72/519654157@392bf8fe
   [junit4]   2> 783261 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 783265 INFO  (zkCallback-520-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6a96d1a1 name:ZooKeeperConnection Watcher:127.0.0.1:39279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 783265 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 783268 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 783270 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 783275 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 783277 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 783279 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 783281 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 783284 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 783285 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 783286 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 783287 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 783288 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 783289 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 783292 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 783294 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 783296 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 783297 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96011522179792900-127.0.0.1:43502_-n_0000000000
   [junit4]   2> 783297 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:43502_
   [junit4]   2> 783297 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 783300 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.Overseer Overseer (id=96011522179792900-127.0.0.1:43502_-n_0000000000) starting
   [junit4]   2> 783302 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 783311 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 783312 INFO  (OverseerCollectionConfigSetProcessor-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 783313 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 783323 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 783325 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43502_
   [junit4]   2> 783335 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:43502_
   [junit4]   2> 783342 INFO  (zkCallback-520-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 783343 INFO  (zkCallback-520-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 783402 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 783402 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 783403 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores
   [junit4]   2> 783405 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 783405 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1
   [junit4]   2> 783405 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 783407 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 783407 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 783409 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 783409 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [n:127.0.0.1:43502_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 783409 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:43502",
   [junit4]   2>   "node_name":"127.0.0.1:43502_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 783410 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:43502",
   [junit4]   2>   "node_name":"127.0.0.1:43502_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 783410 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 783411 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 783412 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 783414 INFO  (zkCallback-520-thread-2-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 783416 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 783417 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.ZkController Waiting for coreNodeName for core collection1 in collection control_collection to be assigned
   [junit4]   2> 783417 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 783417 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 783417 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 783417 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 783419 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 783419 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1'
   [junit4]   2> 783419 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 783419 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 783422 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 783431 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 783433 WARN  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 783433 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 783439 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 783448 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 783583 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 783593 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 783595 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 783616 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 783619 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 783624 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 783626 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 783630 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 783630 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 783630 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 783631 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1/data/]
   [junit4]   2> 783631 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@45679614
   [junit4]   2> 783632 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1/data
   [junit4]   2> 783632 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1/data/index/
   [junit4]   2> 783632 WARN  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 783632 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1/data/index
   [junit4]   2> 783633 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1871910151, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 783633 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6d0e0d44 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@583ba84f),segFN=segments_1,generation=1}
   [junit4]   2> 783633 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 783635 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1/data/
   [junit4]   2> 783636 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 783641 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 783642 WARN  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 783645 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 783647 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 783648 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 783649 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 783652 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/dump,/update/csv,/sql,/graph,/admin/segments,/admin/system,/replication,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file
   [junit4]   2> 783652 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 783653 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 783653 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 783666 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 783666 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 783667 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=35, maxMergedSegmentMB=43.0205078125, floorSegmentMB=0.8798828125, forceMergeDeletesPctAllowed=19.214881646836844, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 783667 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6d0e0d44 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@583ba84f),segFN=segments_1,generation=1}
   [junit4]   2> 783667 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 783667 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@93ac793[collection1] main]
   [junit4]   2> 783668 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 783669 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 783669 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 783670 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 783670 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 783670 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 783671 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 783672 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 783672 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 783672 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 783672 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 783672 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 783672 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 783672 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1536184355894853632
   [junit4]   2> 783673 INFO  (searcherExecutor-1748-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@93ac793[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 783675 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1536184355894853632
   [junit4]   2> 783675 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 783676 INFO  (coreLoadExecutor-1747-thread-1-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 783677 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:43502 collection:control_collection shard:shard1
   [junit4]   2> 783679 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 783685 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/96011522179792900-core_node1-n_0000000000
   [junit4]   2> 783686 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 783690 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 783690 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 783690 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 783690 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:43502/collection1/
   [junit4]   2> 783690 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 783691 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:43502/collection1/ has no replicas
   [junit4]   2> 783693 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 783697 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/96011522179792900-core_node1-n_0000000000
   [junit4]   2> 783699 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:43502/collection1/ shard1
   [junit4]   2> 783700 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:43502",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 783805 INFO  (zkCallback-520-thread-2-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 783854 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:43502/collection1/ and leader is http://127.0.0.1:43502/collection1/
   [junit4]   2> 783854 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43502
   [junit4]   2> 783854 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 783854 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 783855 INFO  (coreZkRegister-1741-thread-1-processing-n:127.0.0.1:43502_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:43502_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 783869 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:43502",
   [junit4]   2>   "node_name":"127.0.0.1:43502_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 783869 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:43502",
   [junit4]   2>   "node_name":"127.0.0.1:43502_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 783921 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:39279/solr
   [junit4]   2> 783926 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 783927 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 783933 INFO  (zkCallback-524-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@35f84ca5 name:ZooKeeperConnection Watcher:127.0.0.1:39279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 783933 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 783933 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 783933 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 783935 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ZkStateReader Loaded empty cluster properties
   [junit4]   2> 783936 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 783938 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 783938 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 783938 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 783939 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 783947 INFO  (zkCallback-525-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4f8fda16 name:ZooKeeperConnection Watcher:127.0.0.1:39279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 783947 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 783947 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 783951 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 2
   [junit4]   2> 783952 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 783954 INFO  (zkCallback-520-thread-2-processing-n:127.0.0.1:43502_) [n:127.0.0.1:43502_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 783955 INFO  (zkCallback-524-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 784064 INFO  (OverseerStateUpdate-96011522179792900-127.0.0.1:43502_-n_0000000000) [n:127.0.0.1:43502_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 784226 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001/cores/collection1
   [junit4]   2> 784228 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001
   [junit4]   2> 784230 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 784235 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5244b43{/,null,AVAILABLE}
   [junit4]   2> 784236 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.e.j.s.ServerConnector Started ServerConnector@3f10dc3{HTTP/1.1,[http/1.1]}{127.0.0.1:62680}
   [junit4]   2> 784237 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.e.j.s.Server Started @789072ms
   [junit4]   2> 784237 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=60584, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001/cores}
   [junit4]   2> 784238 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 784239 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001'
   [junit4]   2> 784239 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 784239 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 784239 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 784240 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 784245 INFO  (zkCallback-526-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1b188524 name:ZooKeeperConnection Watcher:127.0.0.1:39279/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 784245 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 784245 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 784248 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 784248 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001/solr.xml
   [junit4]   2> 784256 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001/cores
   [junit4]   2> 784256 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.CoreContainer New CoreContainer 723100550
   [junit4]   2> 784256 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001]
   [junit4]   2> 784256 WARN  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_78DB6A8408AAFCFC-001/shard-1-001/lib
   [junit4]   2> 784257 INFO  (TEST-LeaderFailoverAfterPartitionTest.test-seed#[78DB6A8408AAFCFC]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 

[...truncated too long message...]


   [junit4]   2> 	4	/solr/clusterprops.json
   [junit4]   2> 	3	/solr/security.json
   [junit4]   2> 	3	/solr/configs/conf1
   [junit4]   2> 	2	/solr/collections/deletereplica_test/state.json
   [junit4]   2> 	2	/solr/collections/collection1/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/clusterstate.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	4	/solr/live_nodes
   [junit4]   2> 	4	/solr/collections
   [junit4]   2> 	2	/solr/overseer/queue-work
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.method=deleteReplicaAndVerifyDirectoryCleanup -Dtests.seed=78DB6A8408AAFCFC -Dtests.slow=true -Dtests.locale=en-GB -Dtests.timezone=Australia/Canberra -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR    190s J0 | DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:43599/gcm: create the collection time out:180s
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([78DB6A8408AAFCFC:83BEE3E8CE555F7]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:606)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1596)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1553)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1608)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1549)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaAndVerifyDirectoryCleanup(DeleteReplicaTest.java:179)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2746884 INFO  (SUITE-DeleteReplicaTest-seed#[78DB6A8408AAFCFC]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest_78DB6A8408AAFCFC-001
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=4145, maxDocsPerChunk=10, blockSize=1), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=4145, blockSize=1)), sim=ClassicSimilarity, locale=en-GB, timezone=Australia/Canberra
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=3,threads=1,free=133755792,total=444776448
   [junit4]   2> NOTE: All tests run in this JVM: [TestReqParamsAPI, TestExceedMaxTermLength, TestManagedResourceStorage, DistributedExpandComponentTest, DistributedQueryElevationComponentTest, TestXIncludeConfig, TestCollectionAPI, TestAddFieldRealTimeGet, TestRandomDVFaceting, TestTestInjection, CurrencyFieldXmlFileTest, CloudExitableDirectoryReaderTest, TestSchemaSimilarityResource, TestCSVLoader, TestExactSharedStatsCache, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestHighlightDedupGrouping, ConnectionManagerTest, TestSmileRequest, ReplicationFactorTest, TestManagedSchemaAPI, TestPostingsSolrHighlighter, DirectUpdateHandlerTest, BufferStoreTest, DocValuesTest, TestLRUCache, HdfsUnloadDistributedZkTest, CdcrReplicationDistributedZkTest, ClusterStateTest, TestRemoteStreaming, TestFieldCollectionResource, HardAutoCommitTest, TestFieldTypeCollectionResource, HdfsLockFactoryTest, TestFreeTextSuggestions, TestSubQueryTransformer, CircularListTest, TestSurroundQueryParser, SchemaVersionSpecificBehaviorTest, RankQueryTest, SolrRequestParserTest, AnalyticsMergeStrategyTest, JsonLoaderTest, TestHashQParserPlugin, SpatialHeatmapFacetsTest, TestFieldCache, TestCollapseQParserPlugin, OverseerCollectionConfigSetProcessorTest, TestBulkSchemaAPI, CursorPagingTest, SyncSliceTest, TestSolrConfigHandlerConcurrent, SimplePostToolTest, DistributedSpellCheckComponentTest, DistributedVersionInfoTest, HdfsBasicDistributedZk2Test, TestJsonRequest, TestRTimerTree, TestFaceting, TestStressLucene, TestSweetSpotSimilarityFactory, XsltUpdateRequestHandlerTest, StressHdfsTest, TestConfigReload, DirectoryFactoryTest, SuggesterFSTTest, TestLeaderElectionZkExpiry, DirectSolrSpellCheckerTest, InfoHandlerTest, HighlighterMaxOffsetTest, ConcurrentDeleteAndCreateCollectionTest, OverseerStatusTest, TestDistributedSearch, TestMiniSolrCloudClusterSSL, TestStressReorder, CursorMarkTest, TestPerFieldSimilarityWithDefaultOverride, TestLRUStatsCache, TestDownShardTolerantSearch, AnalysisAfterCoreReloadTest, SharedFSAutoReplicaFailoverTest, TestComponentsName, TestPhraseSuggestions, TestUseDocValuesAsStored, TestConfig, TestCustomSort, UpdateParamsTest, TestPKIAuthenticationPlugin, TestMiniSolrCloudClusterBase, ExternalFileFieldSortTest, RegexBoostProcessorTest, ActionThrottleTest, TestUninvertingReader, TestQuerySenderNoQuery, TestPartialUpdateDeduplication, TestWordDelimiterFilterFactory, TestCloudBackupRestore, TestSha256AuthenticationProvider, SolrTestCaseJ4Test, IndexSchemaTest, PathHierarchyTokenizerFactoryTest, StatelessScriptUpdateProcessorFactoryTest, SimpleMLTQParserTest, TestLuceneMatchVersion, TestFieldCacheWithThreads, TestBulkSchemaConcurrent, AnalysisErrorHandlingTest, RecoveryAfterSoftCommitTest, TestAnalyzedSuggestions, XmlUpdateRequestHandlerTest, TestRecoveryHdfs, HdfsRestartWhileUpdatingTest, TestReplicationHandler, TestFastLRUCache, DateRangeFieldTest, ZkStateReaderTest, TestIntervalFaceting, TestShardHandlerFactory, TestManagedStopFilterFactory, TestSQLHandler, RequestLoggingTest, TestElisionMultitermQuery, IndexBasedSpellCheckerTest, HdfsThreadLeakTest, TestCollationFieldDocValues, DistributedFacetPivotSmallTest, TestCloudInspectUtil, TestXmlQParser, TestScoreJoinQPNoScore, TestShortCircuitedRequests, CollectionStateFormat2Test, BadComponentTest, TestIBSimilarityFactory, TestDocSet, DistribCursorPagingTest, TestUpdate, ReturnFieldsTest, TestMergePolicyConfig, TestExtendedDismaxParser, TestStressUserVersions, ZkControllerTest, DistributedQueryComponentCustomSortTest, TestExpandComponent, ExplicitHLLTest, TestNoOpRegenerator, TestCryptoKeys, BasicDistributedZkTest, UnloadDistributedZkTest, LeaderElectionIntegrationTest, RecoveryZkTest, ClusterStateUpdateTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, ZkCLITest, AutoCommitTest, TestDistributedGrouping, TestStressVersions, DistributedTermsComponentTest, TestSolr4Spatial, SpellCheckComponentTest, TestFiltering, TestLazyCores, CurrencyFieldOpenExchangeTest, SolrIndexSplitterTest, TestCoreDiscovery, SpatialFilterTest, PolyFieldTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestAtomicUpdateErrorCases, QueryEqualityTest, DefaultValueUpdateProcessorTest, DisMaxRequestHandlerTest, TestWriterPerf, RequestHandlersTest, TestValueSourceCache, RequiredFieldsTest, TestArbitraryIndexDir, SolrPluginUtilsTest, TestCollationField, JSONWriterTest, PingRequestHandlerTest, HighlighterConfigTest, SOLR749Test, TestQuerySenderListener, AlternateDirectoryTest, TestSolrIndexConfig, SolrIndexConfigTest, TestStressRecovery, PreAnalyzedFieldTest, PrimUtilsTest, TestSuggestSpellingConverter, DateFieldTest, DOMUtilTest, RAMDirectoryFactoryTest, TestSolrJ, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, UUIDFieldTest, DistributedIntervalFacetingTest, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, AliasIntegrationTest, BaseCdcrDistributedZkTest, CdcrRequestHandlerTest, CollectionReloadTest, CollectionsAPISolrJTest, DeleteLastCustomShardedReplicaTest, DeleteReplicaTest]
   [junit4] Completed [455/612 (4!)] on J0 in 211.85s, 2 tests, 1 error <<< FAILURES!

[...truncated 522 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:740: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:531: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1427: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2560 tests, 1 error, 3 failures, 88 ignored (73 assumptions) [seed: 78DB6A8408AAFCFC]

Total time: 84 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 626 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/626/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CleanupOldIndexTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.CleanupOldIndexTest:     1) Thread[id=17207, name=OverseerHdfsCoreFailoverThread-96009627917090827-127.0.0.1:61881_solr-n_0000000002, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.CleanupOldIndexTest: 
   1) Thread[id=17207, name=OverseerHdfsCoreFailoverThread-96009627917090827-127.0.0.1:61881_solr-n_0000000002, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E3655CB75050FC5E]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CleanupOldIndexTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=17207, name=OverseerHdfsCoreFailoverThread-96009627917090827-127.0.0.1:61881_solr-n_0000000002, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.interrupt0(Native Method)         at java.lang.Thread.interrupt(Thread.java:923)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=17207, name=OverseerHdfsCoreFailoverThread-96009627917090827-127.0.0.1:61881_solr-n_0000000002, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
        at java.lang.Thread.interrupt0(Native Method)
        at java.lang.Thread.interrupt(Thread.java:923)
        at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E3655CB75050FC5E]:0)




Build Log:
[...truncated 11854 lines...]
   [junit4] Suite: org.apache.solr.cloud.CleanupOldIndexTest
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/init-core-data-001
   [junit4]   2> 1846521 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 1846523 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1846523 INFO  (Thread-5364) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1846523 INFO  (Thread-5364) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1846623 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:41071
   [junit4]   2> 1846623 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1846624 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846627 INFO  (zkCallback-2836-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7025523 name:ZooKeeperConnection Watcher:127.0.0.1:41071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846628 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846628 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1846628 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 1846632 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /solr/clusterprops.json
   [junit4]   2> 1846643 INFO  (jetty-launcher-2835-thread-1) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1846645 INFO  (jetty-launcher-2835-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@9ca5044{/solr,null,AVAILABLE}
   [junit4]   2> 1846647 INFO  (jetty-launcher-2835-thread-1) [    ] o.e.j.s.ServerConnector Started ServerConnector@1b775381{SSL,[ssl, http/1.1]}{127.0.0.1:61881}
   [junit4]   2> 1846647 INFO  (jetty-launcher-2835-thread-1) [    ] o.e.j.s.Server Started @1850772ms
   [junit4]   2> 1846647 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61881}
   [junit4]   2> 1846647 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1846647 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1'
   [junit4]   2> 1846648 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1846648 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1846648 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1846649 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846655 INFO  (zkCallback-2837-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@206c4027 name:ZooKeeperConnection Watcher:127.0.0.1:41071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846655 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846655 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1846656 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1846662 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/.
   [junit4]   2> 1846663 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.CoreContainer New CoreContainer 1962658636
   [junit4]   2> 1846663 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1]
   [junit4]   2> 1846663 WARN  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/lib
   [junit4]   2> 1846663 INFO  (jetty-launcher-2835-thread-2) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1846664 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1846666 INFO  (jetty-launcher-2835-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@36b1710f{/solr,null,AVAILABLE}
   [junit4]   2> 1846667 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 1846668 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1846668 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1846668 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41071/solr
   [junit4]   2> 1846668 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1846668 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1846669 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846669 INFO  (jetty-launcher-2835-thread-2) [    ] o.e.j.s.ServerConnector Started ServerConnector@34b6ff6b{SSL,[ssl, http/1.1]}{127.0.0.1:51127}
   [junit4]   2> 1846670 INFO  (jetty-launcher-2835-thread-2) [    ] o.e.j.s.Server Started @1850795ms
   [junit4]   2> 1846670 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51127}
   [junit4]   2> 1846670 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@6d06d69c
   [junit4]   2> 1846670 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2'
   [junit4]   2> 1846670 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1846670 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1846670 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1846671 INFO  (zkCallback-2840-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@22b5c7c1 name:ZooKeeperConnection Watcher:127.0.0.1:41071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846671 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846671 INFO  (jetty-launcher-2835-thread-1) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1846671 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846673 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$68/829276264@2a55b92a
   [junit4]   2> 1846674 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846674 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x155183f77040002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1846676 INFO  (zkCallback-2842-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@58f41d04 name:ZooKeeperConnection Watcher:127.0.0.1:41071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846676 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846677 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1846677 INFO  (zkCallback-2841-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@127febe6 name:ZooKeeperConnection Watcher:127.0.0.1:41071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846678 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846678 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1846678 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 1846680 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1846683 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/.
   [junit4]   2> 1846683 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.CoreContainer New CoreContainer 1681088118
   [junit4]   2> 1846683 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2]
   [junit4]   2> 1846684 WARN  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/lib
   [junit4]   2> 1846684 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 1846685 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1846685 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 1846686 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1846687 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1846687 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41071/solr
   [junit4]   2> 1846687 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1846687 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1846688 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846688 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1846689 INFO  (zkCallback-2845-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@79586479 name:ZooKeeperConnection Watcher:127.0.0.1:41071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846689 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846690 INFO  (jetty-launcher-2835-thread-2) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1846691 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1846692 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$68/829276264@26a19ebd
   [junit4]   2> 1846693 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846694 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:51127_solr) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6d448d06 name:ZooKeeperConnection Watcher:127.0.0.1:41071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846694 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846694 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1846697 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1846699 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1846702 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1846704 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1846706 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1846706 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1846708 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader Loaded cluster properties: {urlScheme=https}
   [junit4]   2> 1846708 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1846709 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1846716 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader Loaded cluster properties: {urlScheme=https}
   [junit4]   2> 1846717 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1846719 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1846722 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1846723 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1846724 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1846725 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96009627917090819-127.0.0.1:61881_solr-n_0000000000
   [junit4]   2> 1846725 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1846725 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:61881_solr
   [junit4]   2> 1846726 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1846726 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/96009627917090822-127.0.0.1:51127_solr-n_0000000001
   [junit4]   2> 1846727 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/96009627917090819-127.0.0.1:61881_solr-n_0000000000 to know if I could be the leader
   [junit4]   2> 1846728 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.Overseer Overseer (id=96009627917090819-127.0.0.1:61881_solr-n_0000000000) starting
   [junit4]   2> 1846728 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51127_solr
   [junit4]   2> 1846729 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:51127_solr
   [junit4]   2> 1846731 INFO  (zkCallback-2842-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1846731 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:51127_solr) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1846731 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1846733 INFO  (zkCallback-2842-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1846735 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:51127_solr) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1846742 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1846743 INFO  (OverseerCollectionConfigSetProcessor-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1846744 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1846745 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.ZkController Publish node=127.0.0.1:61881_solr as DOWN
   [junit4]   2> 1846747 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61881_solr
   [junit4]   2> 1846748 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 1846748 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:61881_solr
   [junit4]   2> 1846749 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:61881_solr"} current state version: 0
   [junit4]   2> 1846749 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:61881_solr
   [junit4]   2> 1846750 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:51127_solr) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1846750 INFO  (zkCallback-2842-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1846751 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:51127_solr) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1846752 INFO  (zkCallback-2842-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1846787 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1846788 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1846788 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1846788 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1846789 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/.
   [junit4]   2> 1846789 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/.
   [junit4]   2> 1846789 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1846789 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions
   [junit4]   2> 1846790 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 1846790 INFO  (jetty-launcher-2835-thread-1) [n:127.0.0.1:61881_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1846791 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 1846791 INFO  (jetty-launcher-2835-thread-2) [n:127.0.0.1:51127_solr    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1846791 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1846792 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846793 INFO  (zkCallback-2849-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@67d786c5 name:ZooKeeperConnection Watcher:127.0.0.1:41071 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846794 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846794 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1846795 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:41071/solr
   [junit4]   2> 1846797 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1846797 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1846799 INFO  (zkCallback-2852-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@18be4bd6 name:ZooKeeperConnection Watcher:127.0.0.1:41071/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1846799 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1846799 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1846799 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1846800 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ZkStateReader Loaded cluster properties: {urlScheme=https}
   [junit4]   2> 1846801 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 1846804 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1846808 INFO  (SUITE-CleanupOldIndexTest-seed#[E3655CB75050FC5E]-worker) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1846830 INFO  (TEST-CleanupOldIndexTest.test-seed#[E3655CB75050FC5E]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 1846895 INFO  (qtp1936010028-17077) [n:127.0.0.1:51127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params async=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&replicationFactor=2&collection.configName=conf1&name=oldindextest&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1846898 INFO  (qtp1936010028-17077) [n:127.0.0.1:51127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&replicationFactor=2&collection.configName=conf1&name=oldindextest&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1846901 INFO  (OverseerCollectionConfigSetProcessor-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running/mn-eb0adf2a-3e07-4063-b940-cb67b46bdcc5
   [junit4]   2> 1846901 INFO  (qtp1936010028-17082) [n:127.0.0.1:51127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1846904 INFO  (qtp1936010028-17082) [n:127.0.0.1:51127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1846905 INFO  (OverseerCollectionConfigSetProcessor-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>   "name":"oldindextest",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "async":"eb0adf2a-3e07-4063-b940-cb67b46bdcc5",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 1846906 INFO  (OverseerThreadFactory-7909-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : create , {
   [junit4]   2>   "name":"oldindextest",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "async":"eb0adf2a-3e07-4063-b940-cb67b46bdcc5",
   [junit4]   2>   "operation":"create"}
   [junit4]   2> 1846907 INFO  (OverseerThreadFactory-7909-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerCollectionMessageHandler Create collection oldindextest
   [junit4]   2> 1846907 INFO  (OverseerThreadFactory-7909-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerCollectionMessageHandler creating collections conf node /collections/oldindextest 
   [junit4]   2> 1846908 INFO  (OverseerThreadFactory-7909-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.SolrZkClient makePath: /collections/oldindextest
   [junit4]   2> 1846911 INFO  (zkCallback-2852-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 1846911 INFO  (zkCallback-2842-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 1846911 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:51127_solr) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 1846918 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "name":"oldindextest",
   [junit4]   2>   "fromApi":"true",
   [junit4]   2>   "replicationFactor":"2",
   [junit4]   2>   "collection.configName":"conf1",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "stateFormat":"2",
   [junit4]   2>   "async":"eb0adf2a-3e07-4063-b940-cb67b46bdcc5",
   [junit4]   2>   "operation":"create"} current state version: 0
   [junit4]   2> 1846918 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.o.ClusterStateMutator building a new cName: oldindextest
   [junit4]   2> 1846920 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/oldindextest/state.json
   [junit4]   2> 1847006 INFO  (qtp1936010028-17083) [n:127.0.0.1:51127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1847009 INFO  (qtp1936010028-17083) [n:127.0.0.1:51127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 1847016 INFO  (OverseerThreadFactory-7909-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating SolrCores for new collection oldindextest, shardNames [shard1] , replicationFactor : 2
   [junit4]   2> 1847016 INFO  (OverseerThreadFactory-7909-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating core oldindextest_shard1_replica2 as part of shard shard1 of collection oldindextest on 127.0.0.1:61881_solr
   [junit4]   2> 1847016 INFO  (OverseerThreadFactory-7909-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.OverseerCollectionMessageHandler Creating core oldindextest_shard1_replica1 as part of shard shard1 of collection oldindextest on 127.0.0.1:51127_solr
   [junit4]   2> 1847028 INFO  (qtp1936010028-17080) [n:127.0.0.1:51127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={async=eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247&qt=/admin/cores&collection.configName=conf1&name=oldindextest_shard1_replica1&action=CREATE&numShards=1&collection=oldindextest&shard=shard1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1847028 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr    ] o.a.s.h.a.CoreAdminOperation core create command async=eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247&qt=/admin/cores&collection.configName=conf1&name=oldindextest_shard1_replica1&action=CREATE&numShards=1&collection=oldindextest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1847028 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=oldindextest_shard1_replica1, collection.configName=conf1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=oldindextest, numShards=1, dataDir=data/, shard=shard1}
   [junit4]   2> 1847029 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1847031 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.ZkController Registering watch for collection oldindextest
   [junit4]   2> 1847032 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.c.ZkStateReader Add data for [oldindextest] ver [0]
   [junit4]   2> 1847032 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.ZkController Waiting for coreNodeName for core oldindextest_shard1_replica1 in collection oldindextest to be assigned
   [junit4]   2> 1847036 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"oldindextest_shard1_replica1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:51127/solr",
   [junit4]   2>   "node_name":"127.0.0.1:51127_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"oldindextest",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1847036 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"oldindextest_shard1_replica1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:51127/solr",
   [junit4]   2>   "node_name":"127.0.0.1:51127_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"oldindextest",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1847054 INFO  (qtp360930442-17074) [n:127.0.0.1:61881_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={async=eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424&qt=/admin/cores&collection.configName=conf1&name=oldindextest_shard1_replica2&action=CREATE&numShards=1&collection=oldindextest&shard=shard1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1847055 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr    ] o.a.s.h.a.CoreAdminOperation core create command async=eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424&qt=/admin/cores&collection.configName=conf1&name=oldindextest_shard1_replica2&action=CREATE&numShards=1&collection=oldindextest&shard=shard1&wt=javabin&version=2
   [junit4]   2> 1847055 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=oldindextest_shard1_replica2, collection.configName=conf1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=oldindextest, numShards=1, dataDir=data/, shard=shard1}
   [junit4]   2> 1847057 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1847059 INFO  (qtp360930442-17062) [n:127.0.0.1:61881_solr    ] o.a.s.h.a.CoreAdminOperation Checking request status for : eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424
   [junit4]   2> 1847059 INFO  (qtp360930442-17062) [n:127.0.0.1:61881_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 1847061 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.ZkController Registering watch for collection oldindextest
   [junit4]   2> 1847062 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"oldindextest_shard1_replica2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:61881/solr",
   [junit4]   2>   "node_name":"127.0.0.1:61881_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"oldindextest",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1847062 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"oldindextest_shard1_replica2",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:61881/solr",
   [junit4]   2>   "node_name":"127.0.0.1:61881_solr",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"oldindextest",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1847062 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.c.ZkStateReader Add data for [oldindextest] ver [0]
   [junit4]   2> 1847063 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.ZkController Waiting for coreNodeName for core oldindextest_shard1_replica2 in collection oldindextest to be assigned
   [junit4]   2> 1847110 INFO  (qtp1936010028-17080) [n:127.0.0.1:51127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1847112 INFO  (qtp1936010028-17080) [n:127.0.0.1:51127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 1847165 INFO  (OverseerStateUpdate-96009627917090819-127.0.0.1:61881_solr-n_0000000000) [n:127.0.0.1:61881_solr    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/oldindextest/state.json version: 0
   [junit4]   2> 1847166 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:51127_solr) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/oldindextest/state.json] for collection [oldindextest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1847166 INFO  (zkCallback-2842-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/oldindextest/state.json] for collection [oldindextest] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1847167 INFO  (zkCallback-2846-thread-1-processing-n:127.0.0.1:51127_solr) [n:127.0.0.1:51127_solr    ] o.a.s.c.c.ZkStateReader Updating data for [oldindextest] from [0] to [1]
   [junit4]   2> 1847167 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.ZkController waiting to find shard id in clusterstate for oldindextest_shard1_replica1
   [junit4]   2> 1847167 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.ZkController Check for collection zkNode:oldindextest
   [junit4]   2> 1847168 INFO  (zkCallback-2842-thread-1-processing-n:127.0.0.1:61881_solr) [n:127.0.0.1:61881_solr    ] o.a.s.c.c.ZkStateReader Updating data for [oldindextest] from [0] to [1]
   [junit4]   2> 1847168 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.ZkController waiting to find shard id in clusterstate for oldindextest_shard1_replica2
   [junit4]   2> 1847168 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.ZkController Check for collection zkNode:oldindextest
   [junit4]   2> 1847169 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1847169 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/oldindextest]
   [junit4]   2> 1847171 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.c.ZkStateReader path=[/collections/oldindextest] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1847171 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/oldindextest_shard1_replica1'
   [junit4]   2> 1847171 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1847171 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/oldindextest]
   [junit4]   2> 1847171 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1847171 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1847174 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.c.ZkStateReader path=[/collections/oldindextest] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1847174 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/oldindextest_shard1_replica2'
   [junit4]   2> 1847174 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1847174 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1847174 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1847177 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1847179 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1847181 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1847182 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1847186 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1847186 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 1847191 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1847195 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.s.IndexSchema [oldindextest_shard1_replica2] Schema name=cloud-dynamic
   [junit4]   2> 1847213 INFO  (qtp1936010028-17077) [n:127.0.0.1:51127_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :requeststatus with params requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&action=REQUESTSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 1847215 INFO  (qtp1936010028-17077) [n:127.0.0.1:51127_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={requestid=eb0adf2a-3e07-4063-b940-cb67b46bdcc5&action=REQUESTSTATUS&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 1847215 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.s.IndexSchema [oldindextest_shard1_replica1] Schema name=cloud-dynamic
   [junit4]   2> 1847251 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1847251 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1847256 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1847256 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1847257 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 1847258 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'oldindextest_shard1_replica2' using configuration from collection oldindextest
   [junit4]   2> 1847258 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1847258 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore [[oldindextest_shard1_replica2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/oldindextest_shard1_replica2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/./oldindextest_shard1_replica2/data/]
   [junit4]   2> 1847258 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore JMX monitoring not detected for core: oldindextest_shard1_replica2
   [junit4]   2> 1847258 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/./oldindextest_shard1_replica2/data
   [junit4]   2> 1847259 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/./oldindextest_shard1_replica2/data/index/
   [junit4]   2> 1847259 WARN  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore [oldindextest_shard1_replica2] Solr index directory '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/./oldindextest_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1847259 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/./oldindextest_shard1_replica2/data/index
   [junit4]   2> 1847260 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 1847260 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1  x:oldindextest_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'oldindextest_shard1_replica1' using configuration from collection oldindextest
   [junit4]   2> 1847260 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1847260 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrCore [[oldindextest_shard1_replica1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/oldindextest_shard1_replica1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/./oldindextest_shard1_replica1/data/]
   [junit4]   2> 1847261 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrCore JMX monitoring not detected for core: oldindextest_shard1_replica1
   [junit4]   2> 1847261 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/./oldindextest_shard1_replica1/data
   [junit4]   2> 1847261 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/./oldindextest_shard1_replica1/data/index/
   [junit4]   2> 1847261 WARN  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrCore [oldindextest_shard1_replica1] Solr index directory '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/./oldindextest_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1847261 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/./oldindextest_shard1_replica1/data/index
   [junit4]   2> 1847262 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@64bca7b9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6ca79b86),segFN=segments_1,generation=1}
   [junit4]   2> 1847262 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1847263 INFO  (OldIndexDirectoryCleanupThreadForCore-oldindextest_shard1_replica1) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core oldindextest_shard1_replica1 in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/./oldindextest_shard1_replica1/data/
   [junit4]   2> 1847263 WARN  (OldIndexDirectoryCleanupThreadForCore-oldindextest_shard1_replica1) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.DirectoryFactory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node2/./oldindextest_shard1_replica1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1847264 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1847266 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@9ccddaa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@357ec6d4),segFN=segments_1,generation=1}
   [junit4]   2> 1847266 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1847266 INFO  (OldIndexDirectoryCleanupThreadForCore-oldindextest_shard1_replica2) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore Looking for old index directories to cleanup for core oldindextest_shard1_replica2 in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/./oldindextest_shard1_replica2/data/
   [junit4]   2> 1847266 WARN  (OldIndexDirectoryCleanupThreadForCore-oldindextest_shard1_replica2) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.DirectoryFactory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.CleanupOldIndexTest_E3655CB75050FC5E-001/tempDir-001/node1/./oldindextest_shard1_replica2/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1847269 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1847270 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1847271 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1847272 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1847272 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1847273 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1847273 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1847274 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1847274 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1847276 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/graph,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1847276 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1847276 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/graph,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1847276 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1847276 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1847276 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1847277 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1847277 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1847277 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1847277 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@64bca7b9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6ca79b86),segFN=segments_1,generation=1}
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@9ccddaa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@357ec6d4),segFN=segments_1,generation=1}
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@401adc24[oldindextest_shard1_replica1] main]
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@298ee451[oldindextest_shard1_replica2] main]
   [junit4]   2> 1847278 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/oldindextest]
   [junit4]   2> 1847285 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/oldindextest]
   [junit4]   2> 1847287 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.c.c.ZkStateReader path=[/collections/oldindextest] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1847287 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1847288 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.c.ZkStateReader path=[/collections/oldindextest] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1847288 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1847289 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1847289 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1847289 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1847289 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1847289 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1847289 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1847290 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1847290 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1847290 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1847290 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1847290 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1847290 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1847290 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1847290 INFO  (parallelCoreAdminExecutor-7910-thread-1-processing-n:127.0.0.1:61881_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1536154047475613696
   [junit4]   2> 1847291 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1847291 INFO  (searcherExecutor-7914-thread-1-processing-x:oldindextest_shard1_replica2 r:core_node2 eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494041424 CREATE n:127.0.0.1:61881_solr s:shard1 c:oldindextest) [n:127.0.0.1:61881_solr c:oldindextest s:shard1 r:core_node2 x:oldindextest_shard1_replica2] o.a.s.c.SolrCore [oldindextest_shard1_replica2] Registered new searcher Searcher@298ee451[oldindextest_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1847291 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:51127_solr c:oldindextest s:shard1 r:core_node1 x:oldindextest_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1847291 INFO  (parallelCoreAdminExecutor-7911-thread-1-processing-n:127.0.0.1:51127_solr eb0adf2a-3e07-4063-b940-cb67b46bdcc52450131494391247 CREATE) [n:127.0.0.1:5112

[...truncated too long message...]

sting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=17207, name=OverseerHdfsCoreFailoverThread-96009627917090827-127.0.0.1:61881_solr-n_0000000002, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]   2>         at java.lang.Thread.interrupt0(Native Method)
   [junit4]   2>         at java.lang.Thread.interrupt(Thread.java:923)
   [junit4]   2>         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {rnd_b=Lucene50(blocksize=128), _version_=PostingsFormat(name=Direct), a_t=PostingsFormat(name=Direct), a_i=Lucene50(blocksize=128), id=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=1598, maxMBSortInHeap=5.034803335975918, sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=sr-CS, timezone=America/Inuvik
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=3,threads=2,free=206225552,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestManagedSynonymFilterFactory, TestRestManager, SortSpecParsingTest, TestConfigSetsAPI, BaseCdcrDistributedZkTest, TestIndexingPerformance, BasicFunctionalityTest, TestSolrQueryResponse, TestOverriddenPrefixQueryForCustomFieldType, ChangedSchemaMergeTest, HdfsTlogReplayBufferedWhileIndexingTest, CachingDirectoryFactoryTest, TestCollationField, TestSolr4Spatial2, AddBlockUpdateTest, CollectionReloadTest, ZkSolrClientTest, TestRestoreCore, TestSubQueryTransformerCrossCore, TestSizeLimitedDistributedMap, TestScoreJoinQPScore, TestAnalyzeInfixSuggestions, TestGeoJSONResponseWriter, HdfsNNFailoverTest, SecurityConfHandlerTest, TestConfigSetsAPIZkFailure, TestSolrQueryParserDefaultOperatorResource, CollectionTooManyReplicasTest, TestInitParams, TestTolerantUpdateProcessorCloud, AutoCommitTest, SortByFunctionTest, TestCharFilters, TestRequestForwarding, ShardSplitTest, SolrCoreCheckLockOnStartupTest, AsyncCallRequestStatusResponseTest, HdfsSyncSliceTest, DistributedMLTComponentTest, CacheHeaderTest, DateMathParserTest, TestDistributedGrouping, TestRecovery, TestHighFrequencyDictionaryFactory, TestDistributedStatsComponentCardinality, TestFuzzyAnalyzedSuggestions, TestSolrQueryParser, TestTrie, TestReloadAndDeleteDocs, DistributedIntervalFacetingTest, TestTrieFacet, TestCrossCoreJoin, TestObjectReleaseTracker, TestReload, OpenCloseCoreStressTest, StatsComponentTest, CollectionsAPIDistributedZkTest, TestBM25SimilarityFactory, TestMacros, FieldMutatingUpdateProcessorTest, TestSolrCLIRunExample, RemoteQueryErrorTest, ConfigSetsAPITest, OverseerTaskQueueTest, TestDynamicFieldResource, DocValuesMissingTest, DateFieldTest, TestSolrConfigHandlerCloud, BadIndexSchemaTest, TestQueryTypes, ResponseHeaderTest, LeaderFailoverAfterPartitionTest, TestDistributedMissingSort, DebugComponentTest, DeleteInactiveReplicaTest, TestReloadDeadlock, MinimalSchemaTest, CopyFieldTest, TestFoldingMultitermQuery, TestSchemaResource, TestFieldCacheReopen, TestIndexSearcher, SpellingQueryConverterTest, TestSolrDeletionPolicy2, TestSSLRandomization, TestDistribIDF, BigEndianAscendingWordDeserializerTest, TestNamedUpdateProcessors, CSVRequestHandlerTest, TestClassicSimilarityFactory, HdfsDirectoryTest, FacetPivotSmallTest, TestCloudPivotFacet, TestReplicationHandlerBackup, DistributedTermsComponentTest, ShardRoutingCustomTest, ResourceLoaderTest, SpellCheckCollatorTest, SolrPluginUtilsTest, TestLeaderInitiatedRecoveryThread, SolrCloudExampleTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestStressCloudBlindAtomicUpdates, TestManagedResource, TestRawResponseWriter, DirectSolrConnectionTest, AnalyticsQueryTest, ShowFileRequestHandlerTest, TestConfigSets, TestSolrIndexConfig, RestartWhileUpdatingTest, VMParamsZkACLAndCredentialsProvidersTest, JavabinLoaderTest, TestPivotHelperCode, UUIDUpdateProcessorFallbackTest, TestSolrJ, TestImplicitCoreProperties, SpatialRPTFieldTypeTest, DeleteReplicaTest, DistributedFacetPivotWhiteBoxTest, MultiThreadedOCPTest, TestQueryUtils, DistributedQueryComponentOptimizationTest, TestSolrConfigHandler, FullHLLTest, DistributedQueueTest, TestCoreDiscovery, DefaultValueUpdateProcessorTest, CreateCollectionCleanupTest, TestStandardQParsers, TestComplexPhraseQParserPlugin, DistributedDebugComponentTest, TestWriterPerf, TestJsonFacets, TestBadConfig, TestJettySolrRunner, TestNonDefinedSimilarityFactory, TestOmitPositions, BitVectorTest, BasicDistributedZk2Test, OverseerRolesTest, TestBinaryField, TestLegacyFieldCache, TestSuggestSpellingConverter, EchoParamsTest, TestFieldResource, TestSchemaManager, UniqFieldsUpdateProcessorFactoryTest, TestMissingGroups, LeaderElectionIntegrationTest, BasicZkTest, FullSolrCloudDistribCmdsTest, TestFaceting, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, ZkControllerTest, TestRealTimeGet, TestStressReorder, TestJoin, TestStressVersions, HardAutoCommitTest, TestSolr4Spatial, SpellCheckComponentTest, DirectUpdateHandlerTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, SuggesterTSTTest, SuggesterTest, TestCSVLoader, PolyFieldTest, WordBreakSolrSpellCheckerTest, QueryEqualityTest, DistanceFunctionTest, XsltUpdateRequestHandlerTest, TestSurroundQueryParser, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, FieldAnalysisRequestHandlerTest, LoggingHandlerTest, IndexSchemaTest, ReturnFieldsTest, TestCSVResponseWriter, TestAnalyzedSuggestions, PingRequestHandlerTest, HighlighterConfigTest, SOLR749Test, TestQuerySenderNoQuery, SolrIndexConfigTest, MultiTermTest, NumericFieldsTest, TestElisionMultitermQuery, OutputWriterTest, TestSolrCoreProperties, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, SpellPossibilityIteratorTest, SynonymTokenizerTest, TestXIncludeConfig, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, TestFastOutputStream, OpenExchangeRatesOrgProviderTest, PluginInfoTest, TestFastLRUCache, TestSystemIdResolver, PrimUtilsTest, RAMDirectoryFactoryTest, ClusterStateTest, TestUtils, ZkNodePropsTest, SliceStateTest, UUIDFieldTest, FileUtilsTest, CircularListTest, TestRTGBase, CursorPagingTest, TestCursorMarkWithoutUniqueKey, TestSimpleTrackingShardHandler, TestTolerantSearch, ConnectionReuseTest, ActionThrottleTest, AliasIntegrationTest, CdcrReplicationHandlerTest, CleanupOldIndexTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CleanupOldIndexTest -Dtests.seed=E3655CB75050FC5E -Dtests.slow=true -Dtests.locale=sr-CS -Dtests.timezone=America/Inuvik -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | CleanupOldIndexTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.CleanupOldIndexTest: 
   [junit4]    >    1) Thread[id=17207, name=OverseerHdfsCoreFailoverThread-96009627917090827-127.0.0.1:61881_solr-n_0000000002, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E3655CB75050FC5E]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=17207, name=OverseerHdfsCoreFailoverThread-96009627917090827-127.0.0.1:61881_solr-n_0000000002, state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
   [junit4]    >         at java.lang.Thread.interrupt0(Native Method)
   [junit4]    >         at java.lang.Thread.interrupt(Thread.java:923)
   [junit4]    >         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:139)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E3655CB75050FC5E]:0)
   [junit4] Completed [452/612 (1!)] on J1 in 45.28s, 1 test, 2 errors <<< FAILURES!

[...truncated 525 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:740: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:531: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1427: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2560 tests, 2 suite-level errors, 645 ignored (67 assumptions) [seed: E3655CB75050FC5E]

Total time: 52 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 625 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/625/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([9B27F3A369EE1FE:E0E8C402A8077156]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:782)
	at org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir(TestArbitraryIndexDir.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=*[count(//doc)=1]
	xml response was: <?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
</response>

	request was:q=id:2&qt=standard&start=0&rows=20&version=2.2
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:775)
	... 41 more




Build Log:
[...truncated 10722 lines...]
   [junit4] Suite: org.apache.solr.core.TestArbitraryIndexDir
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001
   [junit4]   2> 483929 INFO  (SUITE-TestArbitraryIndexDir-seed#[9B27F3A369EE1FE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 483931 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testLoadNewIndexDir
   [junit4]   2> 483931 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.SolrTestCaseJ4 ####initCore
   [junit4]   2> 483931 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 483931 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 483931 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr
   [junit4]   2> 483932 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 483932 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 483951 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 483956 WARN  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 483968 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 483981 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 483986 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.IndexSchema [null] Schema name=test
   [junit4]   2> 484056 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 484056 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 484057 WARN  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 484061 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 484063 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 484065 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 484065 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr
   [junit4]   2> 484065 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr'
   [junit4]   2> 484065 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 484065 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr
   [junit4]   2> 484065 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.CoreContainer New CoreContainer 457896427
   [junit4]   2> 484065 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr]
   [junit4]   2> 484065 WARN  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/lib
   [junit4]   2> 484066 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,
   [junit4]   2> 484069 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: 
   [junit4]   2> 484070 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 484071 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 484071 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 484071 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 484073 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema12.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, dataDir=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001, shard=shard1}
   [junit4]   2> 484073 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 484073 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 484073 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr
   [junit4]   2> 484074 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 484074 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 484094 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 484099 WARN  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 484100 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 484114 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 484118 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 484194 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 484195 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 484195 WARN  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 484201 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 484203 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 484206 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 484206 INFO  (coreLoadExecutor-1956-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1
   [junit4]   2> 484206 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 484206 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/]
   [junit4]   2> 484207 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c51f00
   [junit4]   2> 484207 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001
   [junit4]   2> 484207 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index/
   [junit4]   2> 484207 WARN  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 484208 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index
   [junit4]   2> 484208 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=875687138, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 484308 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(NIOFSDirectory@/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36c94050),segFN=segments_1,generation=1}
   [junit4]   2> 484308 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 484308 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 484312 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 484313 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 484313 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 484313 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 484314 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 484316 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 484316 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 484317 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 484320 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 484322 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,spellCheckCompRH,lazy,/update/json/docs,/admin/luke,/export,spellCheckWithWordbreak_Direct,tvrh,/get,/admin/properties,/update/json,mltrh,/admin/threads,/search-facet-def,/update/csv,/sql,/graph,/search-facet-invariants,/admin/segments,spellCheckCompRH1,/mlt,/admin/system,/replication,dismax,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,spellCheckWithWordbreak,mock,/admin/ping,/update,/admin/file,/terms,spellCheckCompRH_Direct,/debug/dump
   [junit4]   2> 484322 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 484322 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 484322 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 484322 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4435040056698568]
   [junit4]   2> 484323 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(NIOFSDirectory@/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36c94050),segFN=segments_1,generation=1}
   [junit4]   2> 484323 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 484323 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c5f1265[collection1] main]
   [junit4]   2> 484323 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 484323 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 484324 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 484324 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 484334 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 484408 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 484408 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 484409 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 484409 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 484409 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 484409 INFO  (coreLoadExecutor-1956-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 484410 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 484410 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 484411 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 484411 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.SolrTestCaseJ4 ####initCore end
   [junit4]   2> 484411 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 484411 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 484412 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{add=[1]} 0 0
   [junit4]   2> 484412 INFO  (searcherExecutor-1957-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3c5f1265[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 484454 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 484479 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(NIOFSDirectory@/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36c94050),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(NIOFSDirectory@/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@36c94050),segFN=segments_2,generation=2}
   [junit4]   2> 484479 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 484480 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrCore New index directory detected: old=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index/ new=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index_temp
   [junit4]   2> 484482 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5bd16f5[collection1] main]
   [junit4]   2> 484482 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 484483 INFO  (searcherExecutor-1957-thread-1) [    ] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5bd16f5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1)))}
   [junit4]   2> 484483 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=null path=null params={}{commit=} 0 29
   [junit4]   2> 484483 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 484484 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 484484 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader using system property solr.solr.home: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr
   [junit4]   2> 484484 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 484484 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 484510 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 484516 WARN  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 484517 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
   [junit4]   2> 484545 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 484548 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 484608 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 484610 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 484624 WARN  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (2)
   [junit4]   2> 484628 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 484630 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 484633 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 484633 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1
   [junit4]   2> 484634 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test-files/solr/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/]
   [junit4]   2> 484634 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@c51f00
   [junit4]   2> 484636 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index_temp
   [junit4]   2> 484637 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/
   [junit4]   2> 484637 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.DirectoryFactory Found 1 old index directories to clean-up under /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/
   [junit4]   2> 484638 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Cannot delete directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index as it is still being referenced in the cache!
   [junit4]   2> 484638 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.DirectoryFactory Delete old index directory /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index failed.
   [junit4]   2> 484640 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 484641 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 484643 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 484644 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 484646 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 484646 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 484648 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index_temp
   [junit4]   2> 484652 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 484654 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,spellCheckCompRH,lazy,/update/json/docs,/admin/luke,/export,spellCheckWithWordbreak_Direct,tvrh,/get,/admin/properties,/update/json,mltrh,/admin/threads,/search-facet-def,/update/csv,/sql,/graph,/search-facet-invariants,/admin/segments,spellCheckCompRH1,/mlt,/admin/system,/replication,dismax,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,spellCheckWithWordbreak,mock,/admin/ping,/update,/admin/file,/terms,spellCheckCompRH_Direct,/debug/dump
   [junit4]   2> 484654 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 484654 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 484654 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 484655 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@431ed84c[collection1] main]
   [junit4]   2> 484655 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 484655 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 484655 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 484655 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 484657 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 484663 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 484663 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 484664 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.DefaultSolrCoreState Closing old IndexWriter... core=collection1
   [junit4]   2> 484664 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 484664 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 484664 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 484665 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 484665 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=48.462890625, floorSegmentMB=1.60546875, forceMergeDeletesPctAllowed=25.863935835926792, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.11697163040968094
   [junit4]   2> 484665 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 484667 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(NIOFSDirectory@/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index_temp lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@524fc44c),segFN=segments_1,generation=1}
   [junit4]   2> 484667 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 484667 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 484667 WARN  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrCore [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
   [junit4]   2> 484669 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 484673 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 484673 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@65877308[collection1] main]
   [junit4]   2> 484673 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 484673 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.CoreContainer replacing core: collection1
   [junit4]   2> 484673 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@77db4923
   [junit4]   2> 484673 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.u.DirectUpdateHandler2 closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 484674 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 484674 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@431ed84c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1)))}
   [junit4]   2> 484674 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 484676 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 484676 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 484676 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 484677 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 484677 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 484679 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 484680 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=null path=null params={q=id:2&qt=standard&start=0&rows=20&version=2.2} hits=0 status=0 QTime=0
   [junit4]   2> 484681 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 484681 ERROR (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.SolrTestCaseJ4 REQUEST FAILED: xpath=*[count(//doc)=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=id:2&qt=standard&start=0&rows=20&version=2.2
   [junit4]   2> 484681 ERROR (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.SolrTestCaseJ4 REQUEST FAILED: q=id:2&qt=standard&start=0&rows=20&version=2.2:java.lang.RuntimeException: REQUEST FAILED: xpath=*[count(//doc)=1]
   [junit4]   2> 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]   2> <response>
   [junit4]   2> <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]   2> </response>
   [junit4]   2> 
   [junit4]   2> 	request was:q=id:2&qt=standard&start=0&rows=20&version=2.2
   [junit4]   2> 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:775)
   [junit4]   2> 	at org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir(TestArbitraryIndexDir.java:107)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 484682 INFO  (TEST-TestArbitraryIndexDir.testLoadNewIndexDir-seed#[9B27F3A369EE1FE]) [    x:collection1] o.a.s.SolrTestCaseJ4 ###Ending testLoadNewIndexDir
   [junit4]   2> 484683 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestArbitraryIndexDir -Dtests.method=testLoadNewIndexDir -Dtests.seed=9B27F3A369EE1FE -Dtests.slow=true -Dtests.locale=pl-PL -Dtests.timezone=Asia/Omsk -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4]   2> 484684 INFO  (searcherExecutor-1961-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@65877308[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):c1)))}
   [junit4] ERROR   0.75s J0 | TestArbitraryIndexDir.testLoadNewIndexDir <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9B27F3A369EE1FE:E0E8C402A8077156]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:782)
   [junit4]    > 	at org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir(TestArbitraryIndexDir.java:107)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=*[count(//doc)=1]
   [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><result name="response" numFound="0" start="0"></result>
   [junit4]    > </response>
   [junit4]    > 	request was:q=id:2&qt=standard&start=0&rows=20&version=2.2
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:775)
   [junit4]    > 	... 41 more
   [junit4]   2> 484684 INFO  (SUITE-TestArbitraryIndexDir-seed#[9B27F3A369EE1FE]-worker) [    x:collection1] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> 484684 INFO  (SUITE-TestArbitraryIndexDir-seed#[9B27F3A369EE1FE]-worker) [    x:collection1] o.a.s.c.CoreContainer Shutting down CoreContainer instance=457896427
   [junit4]   2> 484685 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@48053f8e
   [junit4]   2> 484685 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.u.DirectUpdateHandler2 closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 484685 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.u.SolrCoreState Closing SolrCoreState
   [junit4]   2> 484685 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.u.DefaultSolrCoreState SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 484685 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.u.DefaultSolrCoreState closing IndexWriter with IndexWriterCloser
   [junit4]   2> 484687 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Closing main searcher on request.
   [junit4]   2> 484706 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing MockFSDirectoryFactory - 3 directories currently being tracked
   [junit4]   2> 484706 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 484706 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 484706 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index
   [junit4]   2> 484709 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index_temp [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index_temp;done=false>>]
   [junit4]   2> 484709 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 484709 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001/index_temp
   [junit4]   2> 484711 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001 [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001;done=false>>]
   [junit4]   2> 484711 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 484711 INFO  (coreCloseExecutor-1965-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.core.TestArbitraryIndexDir_9B27F3A369EE1FE-001
   [junit4]   2> NOTE: test params are: codec=Lucene62, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=pl-PL, timezone=Asia/Omsk
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_92 (64-bit)/cpus=3,threads=1,free=62667472,total=237166592
   [junit4]   2> NOTE: All tests run in this JVM: [TestSweetSpotSimilarityFactory, ClusterStateTest, OverseerCollectionConfigSetProcessorTest, TestDefaultStatsCache, SpatialFilterTest, DistributedFacetPivotSmallTest, SharedFSAutoReplicaFailoverTest, TestFreeTextSuggestions, ZkControllerTest, TestQueryWrapperFilter, PeerSyncTest, TestTrackingShardHandlerFactory, TestSha256AuthenticationProvider, TestSolrConfigHandlerConcurrent, TestBlendedInfixSuggestions, TestManagedSchemaAPI, TestCloudInspectUtil, DirectUpdateHandlerTest, ReturnFieldsTest, TestDownShardTolerantSearch, TermsComponentTest, TestSolrCloudWithKerberosAlt, TestCustomDocTransformer, BlockJoinFacetDistribTest, HdfsRecoverLeaseTest, MoreLikeThisHandlerTest, ConjunctionSolrSpellCheckerTest, PKIAuthenticationIntegrationTest, TestLeaderElectionZkExpiry, TestSearchPerf, SuggesterFSTTest, TestFunctionQuery, UnloadDistributedZkTest, ScriptEngineTest, SolrCoreTest, TolerantUpdateProcessorTest, TestDocumentBuilder, SolrRequestParserTest, TestReplicationHandler, TestRTimerTree, DocValuesTest, HdfsRestartWhileUpdatingTest, TestCodecSupport, TestExpandComponent, MigrateRouteKeyTest, TestConfigOverlay, TestStressLucene, TestConfigSetProperties, BadCopyFieldTest, TestArbitraryIndexDir]
   [junit4] Completed [83/612 (1!)] on J0 in 0.81s, 1 test, 1 error <<< FAILURES!

[...truncated 1691 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:740: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:684: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:531: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1427: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:984: There were test failures: 612 suites (10 ignored), 2560 tests, 1 error, 88 ignored (73 assumptions) [seed: 9B27F3A369EE1FE]

Total time: 75 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any