You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/09/25 09:56:31 UTC

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 638 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/638/

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12191 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/init-core-data-001
   [junit4]   2> 852085 T19477 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 852086 T19477 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 852090 T19477 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 852091 T19477 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 852091 T19478 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 852192 T19477 oasc.ZkTestServer.run start zk server on port:39649
   [junit4]   2> 852193 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 852194 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 852197 T19484 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bf4b016 name:ZooKeeperConnection Watcher:127.0.0.1:39649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 852198 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 852198 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 852198 T19477 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 852201 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 852203 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 852204 T19486 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61f3e788 name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 852204 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 852205 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 852205 T19477 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 852207 T19477 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 852209 T19477 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 852210 T19477 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 852212 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 852213 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 852216 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 852216 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 852320 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 852321 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 852323 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 852323 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 852325 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 852326 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 852328 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 852329 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 852331 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 852332 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 852333 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 852334 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 852336 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 852337 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 852338 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 852339 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 852341 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 852342 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 852344 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 852345 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 852346 T19488 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@524aa681 name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 852346 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 852347 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 852348 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 852349 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 852350 T19490 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@452e24d7 name:ZooKeeperConnection Watcher:127.0.0.1:39649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 852350 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 852350 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 852351 T19477 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 852352 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 852353 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 852354 T19492 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@205de707 name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 852354 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 852354 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 852356 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 852357 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 852359 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 852360 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/schema.xml
   [junit4]   2> 852463 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 852463 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 852465 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 852466 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 852467 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 852468 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/protwords.txt
   [junit4]   2> 852470 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 852470 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/currency.xml
   [junit4]   2> 852472 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 852473 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 852474 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 852475 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 852477 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 852477 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 852479 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 852480 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 852481 T19477 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 852482 T19477 oascc.SolrZkClient.makePath makePath: /configs/conf2/elevate.xml
   [junit4]   2> 854183 T19477 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 854190 T19477 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 854193 T19477 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:39655
   [junit4]   2> 854197 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 854198 T19477 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 854198 T19477 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-002
   [junit4]   2> 854199 T19477 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-002/'
   [junit4]   2> 854254 T19477 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-002/solr.xml
   [junit4]   2> 854365 T19477 oasc.CoreContainer.<init> New CoreContainer 1749421491
   [junit4]   2> 854366 T19477 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-002/]
   [junit4]   2> 854368 T19477 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 854368 T19477 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 854369 T19477 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 854369 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 854370 T19477 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 854370 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 854371 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 854372 T19477 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 854372 T19477 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 854376 T19477 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 854380 T19477 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 854380 T19477 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 854381 T19477 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 854382 T19477 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39649/solr
   [junit4]   2> 854382 T19477 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 854383 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 854384 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 854386 T19503 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a2ea17c name:ZooKeeperConnection Watcher:127.0.0.1:39649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 854387 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 854388 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 854390 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 854392 T19505 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12af17af name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 854392 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 854394 T19477 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 854397 T19477 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 854400 T19477 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 854403 T19477 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 854406 T19477 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 854410 T19477 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 854411 T19477 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39655_
   [junit4]   2> 854412 T19477 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39655_
   [junit4]   2> 854415 T19477 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 854417 T19477 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 854418 T19477 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 854420 T19477 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39655_
   [junit4]   2> 854421 T19477 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 854423 T19477 oasc.Overseer.start Overseer (id=92512414122835974-127.0.0.1:39655_-n_0000000000) starting
   [junit4]   2> 854426 T19477 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 854435 T19477 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 854436 T19507 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 854437 T19477 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 854440 T19477 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 854442 T19477 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 854446 T19506 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 854451 T19477 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 854451 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 854457 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 854458 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 854460 T19511 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2307dd5c name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 854461 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 854461 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 854463 T19477 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 854466 T19477 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 854466 T19477 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 854467 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 854468 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 854470 T19513 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2771407a name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 854470 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 854470 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 854473 T19514 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 854477 T19506 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 854477 T19506 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 854477 T19506 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 854481 T19506 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 854483 T19514 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 854483 T19515 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 856213 T19477 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 856214 T19477 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 856219 T19477 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 856221 T19477 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:44310
   [junit4]   2> 856225 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 856226 T19477 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 856226 T19477 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003
   [junit4]   2> 856226 T19477 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/'
   [junit4]   2> 856299 T19477 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/solr.xml
   [junit4]   2> 856399 T19477 oasc.CoreContainer.<init> New CoreContainer 893886510
   [junit4]   2> 856400 T19477 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/]
   [junit4]   2> 856401 T19477 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 856402 T19477 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 856402 T19477 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 856403 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 856403 T19477 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 856403 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 856404 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 856404 T19477 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 856404 T19477 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 856407 T19477 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 856410 T19477 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 856411 T19477 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 856411 T19477 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 856411 T19477 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39649/solr
   [junit4]   2> 856412 T19477 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 856412 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 856413 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 856415 T19526 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3f10f842 name:ZooKeeperConnection Watcher:127.0.0.1:39649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 856416 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 856417 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 856419 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 856421 T19528 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c824f93 name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 856421 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 856427 T19477 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 857436 T19477 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44310_
   [junit4]   2> 857437 T19477 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44310_
   [junit4]   2> 857443 T19477 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 857449 T19477 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 857449 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 859348 T19477 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 859349 T19477 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 859355 T19477 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 859358 T19477 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:23716
   [junit4]   2> 859362 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 859362 T19477 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 859362 T19477 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004
   [junit4]   2> 859363 T19477 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/'
   [junit4]   2> 859420 T19477 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/solr.xml
   [junit4]   2> 859510 T19477 oasc.CoreContainer.<init> New CoreContainer 1966064132
   [junit4]   2> 859511 T19477 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/]
   [junit4]   2> 859512 T19477 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 859513 T19477 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 859513 T19477 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 859514 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 859514 T19477 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 859514 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 859515 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 859515 T19477 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 859515 T19477 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 859519 T19477 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 859521 T19477 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 859522 T19477 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 859522 T19477 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 859522 T19477 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39649/solr
   [junit4]   2> 859523 T19477 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 859523 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 859524 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 859526 T19541 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17244564 name:ZooKeeperConnection Watcher:127.0.0.1:39649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 859528 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 859529 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 859532 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 859533 T19543 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3cdfdb32 name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 859533 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 859626 T19477 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 860633 T19477 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23716_
   [junit4]   2> 860634 T19477 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23716_
   [junit4]   2> 860641 T19477 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 860649 T19477 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 860650 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 862608 T19477 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 862609 T19477 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 862616 T19477 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 862618 T19477 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:63735
   [junit4]   2> 862622 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 862623 T19477 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 862623 T19477 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005
   [junit4]   2> 862624 T19477 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/'
   [junit4]   2> 862677 T19477 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/solr.xml
   [junit4]   2> 862771 T19477 oasc.CoreContainer.<init> New CoreContainer 1000581878
   [junit4]   2> 862772 T19477 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/]
   [junit4]   2> 862774 T19477 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 862775 T19477 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 862775 T19477 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 862776 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 862776 T19477 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 862776 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 862777 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 862777 T19477 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 862778 T19477 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 862781 T19477 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 862783 T19477 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 862784 T19477 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 862784 T19477 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 862785 T19477 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39649/solr
   [junit4]   2> 862785 T19477 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 862785 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 862786 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 862788 T19556 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@792ec7e6 name:ZooKeeperConnection Watcher:127.0.0.1:39649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 862790 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 862790 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 862793 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 862794 T19558 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2395350f name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 862795 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 862801 T19477 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 863807 T19477 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63735_
   [junit4]   2> 863808 T19477 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63735_
   [junit4]   2> 863816 T19477 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 863822 T19477 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 863823 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 865439 T19477 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 865440 T19477 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 865446 T19477 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 865448 T19477 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:37396
   [junit4]   2> 865451 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 865452 T19477 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 865452 T19477 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006
   [junit4]   2> 865453 T19477 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/'
   [junit4]   2> 865505 T19477 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/solr.xml
   [junit4]   2> 865591 T19477 oasc.CoreContainer.<init> New CoreContainer 1210130804
   [junit4]   2> 865591 T19477 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/]
   [junit4]   2> 865593 T19477 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 865593 T19477 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 865593 T19477 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 865594 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 865594 T19477 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 865594 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 865595 T19477 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 865595 T19477 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 865595 T19477 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 865598 T19477 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 865601 T19477 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 865601 T19477 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 865601 T19477 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 865602 T19477 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39649/solr
   [junit4]   2> 865602 T19477 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 865603 T19477 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 865604 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 865605 T19571 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d5e86c5 name:ZooKeeperConnection Watcher:127.0.0.1:39649 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 865607 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 865607 T19477 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 865609 T19477 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 865611 T19573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@744c254f name:ZooKeeperConnection Watcher:127.0.0.1:39649/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 865611 T19477 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 865617 T19477 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 866624 T19477 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37396_
   [junit4]   2> 866626 T19477 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37396_
   [junit4]   2> 866632 T19477 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 866639 T19477 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 866639 T19477 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 866798 T19520 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&name=solrj_collection&collection.configName=conf1&router.field=myOwnField&action=CREATE&numShards=2&wt=javabin&version=2
   [junit4]   2> 866799 T19514 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 866801 T19507 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 866803 T19577 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 866804 T19577 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 866805 T19577 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 866808 T19514 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 866812 T19506 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 866812 T19506 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 866813 T19506 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 866816 T19574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 866816 T19529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 866816 T19544 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 866816 T19514 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 866816 T19515 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 866816 T19559 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 866909 T19577 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 866910 T19577 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:63735_
   [junit4]   2> 866911 T19577 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:23716_
   [junit4]   2> 866913 T19577 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:44310_
   [junit4]   2> 866917 T19577 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:37396_
   [junit4]   2> 867046 T19548 oasha.CoreAdminHandler.handleCreateAction core create command name=solrj_collection_shard1_replica1&collection.configName=conf1&action=CREATE&shard=shard1&numShards=2&collection=solrj_collection&wt=javabin&version=2&qt=/admin/cores
   [junit4]   2> 867049 T19534 oasha.CoreAdminHandler.handleCreateAction core create command name=solrj_collection_shard1_replica2&collection.configName=conf1&action=CREATE&shard=shard1&numShards=2&collection=solrj_collection&wt=javabin&version=2&qt=/admin/cores
   [junit4]   2> 867049 T19518 oasha.CoreAdminHandler.handleCreateAction core create command name=solrj_collection_shard2_replica1&collection.configName=conf1&action=CREATE&shard=shard2&numShards=2&collection=solrj_collection&wt=javabin&version=2&qt=/admin/cores
   [junit4]   2> 867050 T19548 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 867050 T19563 oasha.CoreAdminHandler.handleCreateAction core create command name=solrj_collection_shard2_replica2&collection.configName=conf1&action=CREATE&shard=shard2&numShards=2&collection=solrj_collection&wt=javabin&version=2&qt=/admin/cores
   [junit4]   2> 867051 T19548 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 867051 T19514 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 867053 T19534 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 867054 T19534 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 867054 T19563 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 867054 T19518 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 867055 T19563 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 867056 T19506 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:63735",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:63735_",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1"}
   [junit4]   2> 867056 T19518 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 867059 T19506 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:23716",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:23716_",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2"}
   [junit4]   2> 867061 T19514 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 867062 T19506 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:37396",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:37396_",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2"}
   [junit4]   2> 867065 T19506 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44310",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:44310_",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1"}
   [junit4]   2> 867067 T19514 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 867068 T19529 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 867068 T19514 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 867068 T19544 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 867068 T19559 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 867068 T19574 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 867068 T19515 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 868053 T19548 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 868054 T19548 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 868055 T19548 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 868055 T19548 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 868055 T19534 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 868056 T19534 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 868056 T19548 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 868057 T19534 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 868057 T19548 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/solrj_collection_shard1_replica1/'
   [junit4]   2> 868057 T19563 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 868057 T19534 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 868058 T19563 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 868058 T19518 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 868059 T19563 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 868059 T19518 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 868059 T19534 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 868060 T19518 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 868060 T19534 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/solrj_collection_shard1_replica2/'
   [junit4]   2> 868059 T19563 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 868060 T19518 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 868062 T19563 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 868063 T19518 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 868064 T19563 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/solrj_collection_shard2_replica2/'
   [junit4]   2> 868064 T19518 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/solrj_collection_shard2_replica1/'
   [junit4]   2> 868126 T19548 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 868133 T19534 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 868145 T19563 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 868147 T19518 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 868191 T19548 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 868197 T19534 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 868207 T19563 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 868222 T19518 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 868294 T19548 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 868299 T19534 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 868307 T19548 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 868310 T19563 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 868314 T19534 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 868325 T19518 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 868337 T19563 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 868343 T19518 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 868921 T19534 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 868938 T19534 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 868941 T19548 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 868943 T19534 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 868950 T19548 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 868951 T19518 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 868954 T19548 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 868958 T19534 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 868961 T19518 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 868965 T19518 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 868965 T19534 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 868972 T19534 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 868973 T19548 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 868974 T19534 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 868977 T19534 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 868978 T19534 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 868979 T19534 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 868979 T19534 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 868980 T19534 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 868980 T19534 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 868981 T19534 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/solrj_collection_shard1_replica2/data/
   [junit4]   2> 868981 T19534 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@44a9674f
   [junit4]   2> 868982 T19563 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 868983 T19518 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 868983 T19534 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/solrj_collection_shard1_replica2/data
   [junit4]   2> 868983 T19548 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 868984 T19534 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 868985 T19534 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 868986 T19534 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-004/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 868986 T19534 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=45.59765625, floorSegmentMB=0.658203125, forceMergeDeletesPctAllowed=22.468454961642934, segmentsPerTier=25.0, maxCFSSegmentSizeMB=0.5390625, noCFSRatio=0.8789252069963214
   [junit4]   2> 868988 T19534 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-57965966)),segFN=segments_1,generation=1}
   [junit4]   2> 868988 T19534 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 868989 T19518 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 868991 T19563 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 868991 T19548 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 868995 T19563 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 868996 T19534 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 868996 T19548 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 868997 T19534 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 868997 T19548 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 868998 T19534 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 868998 T19548 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 868999 T19534 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 869000 T19534 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 869000 T19548 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 869000 T19534 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 869000 T19548 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 869001 T19534 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 869001 T19548 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 869002 T19534 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 869002 T19548 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 869002 T19518 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 869003 T19548 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/solrj_collection_shard1_replica1/data/
   [junit4]   2> 869003 T19534 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 869004 T19518 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 869004 T19548 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@44a9674f
   [junit4]   2> 869005 T19534 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 869005 T19518 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 869006 T19534 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 869006 T19548 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/solrj_collection_shard1_replica1/data
   [junit4]   2> 869007 T19534 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 869006 T19518 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 869008 T19534 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 869007 T19548 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 869009 T19534 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 869009 T19548 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 869009 T19518 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 869010 T19548 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-005/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 869008 T19563 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 869011 T19548 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=45.59765625, floorSegmentMB=0.658203125, forceMergeDeletesPctAllowed=22.468454961642934, segmentsPerTier=25.0, maxCFSSegmentSizeMB=0.5390625, noCFSRatio=0.8789252069963214
   [junit4]   2> 869010 T19518 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 869009 T19534 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 869012 T19518 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 869013 T19534 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 869013 T19548 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-43445e28)),segFN=segments_1,generation=1}
   [junit4]   2> 869014 T19534 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 869013 T19518 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 869015 T19534 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 869015 T19518 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/solrj_collection_shard2_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/solrj_collection_shard2_replica1/data/
   [junit4]   2> 869016 T19534 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 869014 T19548 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 869016 T19518 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@44a9674f
   [junit4]   2> 869017 T19534 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 869018 T19518 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/solrj_collection_shard2_replica1/data
   [junit4]   2> 869018 T19563 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 869019 T19518 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 869020 T19518 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 869021 T19518 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-003/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 869022 T19518 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=45.59765625, floorSegmentMB=0.658203125, forceMergeDeletesPctAllowed=22.468454961642934, segmentsPerTier=25.0, maxCFSSegmentSizeMB=0.5390625, noCFSRatio=0.8789252069963214
   [junit4]   2> 869023 T19548 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 869023 T19518 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-146fd545)),segFN=segments_1,generation=1}
   [junit4]   2> 869024 T19548 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 869024 T19518 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 869025 T19548 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 869026 T19548 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 869026 T19563 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 869026 T19548 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 869027 T19548 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 869027 T19563 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 869028 T19548 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 869028 T19563 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 869029 T19548 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 869029 T19563 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 869029 T19548 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 869030 T19563 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 869031 T19548 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 869031 T19563 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 869032 T19548 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 869032 T19563 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 869032 T19518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 869033 T19548 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 869033 T19563 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 869033 T19518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 869034 T19563 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/solrj_collection_shard2_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/solrj_collection_shard2_replica2/data/
   [junit4]   2> 869034 T19548 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 869035 T19563 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@44a9674f
   [junit4]   2> 869035 T19518 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 869036 T19548 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 869037 T19548 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 869037 T19518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 869038 T19548 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 869037 T19563 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/solrj_collection_shard2_replica2/data
   [junit4]   2> 869039 T19548 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 869038 T19518 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 869040 T19548 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 869040 T19518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 869039 T19563 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 869041 T19518 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 869041 T19563 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 869041 T19548 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 869042 T19563 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001/tempDir-006/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 869042 T19518 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 869043 T19563 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=45.59765625, floorSegmentMB=0.658203125, forceMergeDeletesPctAllowed=22.468454961642934, segmentsPerTier=25.0, maxCFSSegmentSizeMB=0.5390625, noCFSRatio=0.8789252069963214
   [junit4]   2> 869043 T19548 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 869043 T19534 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 869045 T19563 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		comm

[...truncated too long message...]

LE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection0_2
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BB68C1B970D86BDC]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=20042, name=Thread-3934, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection5_2
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BB68C1B970D86BDC]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=20040, name=Thread-3932, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection3_2
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BB68C1B970D86BDC]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=20038, name=Thread-3930, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection1_3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BB68C1B970D86BDC]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=20039, name=Thread-3931, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection2_3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BB68C1B970D86BDC]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]   2> 1759921 T19477 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-BB68C1B970D86BDC-001
   [junit4]   2> 907844 T19476 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {intDefault=Lucene41(blocksize=128), text=PostingsFormat(name=Direct), range_facet_l=OrdsLucene41(blocksize=128), _version_=Lucene41(blocksize=128), multiDefault=OrdsLucene41(blocksize=128), a_t=OrdsLucene41(blocksize=128), id=PostingsFormat(name=Memory doPackFST= true), timestamp=OrdsLucene41(blocksize=128), a_i1=OrdsLucene41(blocksize=128), other_tl1=Lucene41(blocksize=128)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=DefaultSimilarity, locale=da, timezone=Pacific/Midway
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=238994216,total=529006592
   [junit4]   2> NOTE: All tests run in this JVM: [TestExceedMaxTermLength, CoreContainerCoreInitFailuresTest, ReplicationFactorTest, TestFastLRUCache, TestLMJelinekMercerSimilarityFactory, FieldMutatingUpdateProcessorTest, TestInitParams, SolrIndexSplitterTest, AddBlockUpdateTest, SpellCheckComponentTest, TestNonNRTOpen, StressHdfsTest, TermVectorComponentDistributedTest, TestCursorMarkWithoutUniqueKey, SuggesterTSTTest, TestNonDefinedSimilarityFactory, AssignTest, ClusterStateUpdateTest, TestDistributedGrouping, TestImplicitCoreProperties, DocumentAnalysisRequestHandlerTest, TestCoreDiscovery, PluginInfoTest, CoreAdminHandlerTest, TestInfoStreamLogging, SharedFSAutoReplicaFailoverUtilsTest, ZkCLITest, HdfsUnloadDistributedZkTest, TestSolrXmlPersistence, HdfsWriteToMultipleCollectionsTest, CustomCollectionTest, TestMergePolicyConfig, SuggestComponentTest, WordBreakSolrSpellCheckerTest, TestRandomMergePolicy, TermVectorComponentTest, StandardRequestHandlerTest, DeleteShardTest, BadIndexSchemaTest, TestNoOpRegenerator, HdfsBasicDistributedZk2Test, TestShortCircuitedRequests, SolrXmlInZkTest, SearchHandlerTest, StatelessScriptUpdateProcessorFactoryTest, ModifyConfFileTest, TestFastWriter, TriLevelCompositeIdRoutingTest, RequestHandlersTest, TestLRUCache, DocExpirationUpdateProcessorFactoryTest, FieldAnalysisRequestHandlerTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J3 in 908.38s, 1 test, 1 error <<< FAILURES!

[...truncated 735 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 430 suites, 1775 tests, 1 error, 54 ignored (10 assumptions)

Total time: 176 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 707557858 bytes
Compression is 0.0%
Took 2 min 0 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 648 - Still Failing

Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
This is a bug. I opened SOLR-6591.

On Sun, Oct 5, 2014 at 11:46 PM, Apache Jenkins Server <
jenkins@builds.apache.org> wrote:

> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/648/
>
> 2 tests failed.
> REGRESSION:
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch
>
> Error Message:
> Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to
> create core [halfcollection_shard1_replica1] Caused by: Could not get shard
> id for core: halfcollection_shard1_replica1
>
> Stack Trace:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create
> core [halfcollection_shard1_replica1] Caused by: Could not get shard id for
> core: halfcollection_shard1_replica1
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>         at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
>         at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
>         at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> REGRESSION:
> org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch
>
> Error Message:
> IOException occured when talking to server at:
> http://127.0.0.1:63235/collection1
>
> Stack Trace:
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at: http://127.0.0.1:63235/collection1
>         at
> __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF:4ACA444C5732CE93]:0)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:584)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>         at
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
>         at
> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
>         at
> org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
>         at
> org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
>         at
> org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:414)
>         at
> org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:144)
>         at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketException: Connection reset
>         at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
>         at
> org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159)
>         at
> org.apache.http.impl.io.AbstractSessionOutputBuffer.writeLine(AbstractSessionOutputBuffer.java:258)
>         at
> org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:59)
>         at
> org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:46)
>         at
> org.apache.http.impl.io.AbstractMessageWriter.write(AbstractMessageWriter.java:109)
>         at
> org.apache.http.impl.AbstractHttpClientConnection.sendRequestHeader(AbstractHttpClientConnection.java:254)
>         at
> org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader(DefaultClientConnection.java:270)
>         at
> org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestHeader(ManagedClientConnectionImpl.java:209)
>         at
> org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:203)
>         at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
>         at
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
>         at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
>         at
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
>         at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>         at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
>         at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
>         at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:466)
>         ... 49 more
>
>
>
>
> Build Log:
> [...truncated 11952 lines...]
>    [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
>    [junit4]   2> Creating dataDir:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/init-core-data-001
>    [junit4]   2> 463604 T1857
> oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext
> system property: /
>    [junit4]   2> 463613 T1857 oas.SolrTestCaseJ4.setUp ###Starting
> testDistribSearch
>    [junit4]   2> 463614 T1857 oasc.ZkTestServer.run STARTING ZK TEST SERVER
>    [junit4]   1> client port:0.0.0.0/0.0.0.0:0
>    [junit4]   2> 463614 T1858 oasc.ZkTestServer$ZKServerMain.runFromConfig
> Starting server
>    [junit4]   2> 463714 T1857 oasc.ZkTestServer.run start zk server on
> port:63231
>    [junit4]   2> 463715 T1857
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 463716 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 463718 T1864 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@3ab7a30a
> name:ZooKeeperConnection Watcher:127.0.0.1:63231 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 463719 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 463719 T1857 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 463719 T1857 oascc.SolrZkClient.makePath makePath: /solr
>    [junit4]   2> 463721 T1857
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 463722 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 463723 T1866 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@653818d4
> name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 463723 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 463723 T1857 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 463724 T1857 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
>    [junit4]   2> 463726 T1857 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/shards
>    [junit4]   2> 463727 T1857 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection
>    [junit4]   2> 463729 T1857 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/shards
>    [junit4]   2> 463731 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to /configs/conf1/solrconfig.xml
>    [junit4]   2> 463731 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.xml
>    [junit4]   2> 463734 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml
> to /configs/conf1/schema.xml
>    [junit4]   2> 463734 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/schema.xml
>    [junit4]   2> 463736 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 463737 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 463738 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf1/stopwords.txt
>    [junit4]   2> 463739 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/stopwords.txt
>    [junit4]   2> 463741 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf1/protwords.txt
>    [junit4]   2> 463741 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/protwords.txt
>    [junit4]   2> 463743 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml
> to /configs/conf1/currency.xml
>    [junit4]   2> 463743 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/currency.xml
>    [junit4]   2> 463745 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
> to /configs/conf1/enumsConfig.xml
>    [junit4]   2> 463746 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/enumsConfig.xml
>    [junit4]   2> 463747 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 463748 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 463750 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 463750 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 463752 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf1/old_synonyms.txt
>    [junit4]   2> 463752 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/old_synonyms.txt
>    [junit4]   2> 463754 T1857 oasc.AbstractZkTestCase.putConfig put
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf1/synonyms.txt
>    [junit4]   2> 463755 T1857 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/synonyms.txt
>    [junit4]   2> 464885 T1857 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 464889 T1857 oejs.AbstractConnector.doStart Started
> SocketConnector@127.0.0.1:63235
>    [junit4]   2> 464889 T1857 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 464890 T1857 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 464890 T1857 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002
>    [junit4]   2> 464890 T1857 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/'
>    [junit4]   2> 464920 T1857 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/solr.xml
>    [junit4]   2> 464974 T1857 oasc.CoreContainer.<init> New CoreContainer
> 239935073
>    [junit4]   2> 464974 T1857 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/]
>    [junit4]   2> 464975 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 464975 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 464975 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 464977 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 464977 T1857 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 464977 T1857 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 464978 T1857 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 464978 T1857 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 464978 T1857 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:63231/solr
>    [junit4]   2> 464978 T1857 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 464979 T1857
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 464979 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 464981 T1877 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6905799c
> name:ZooKeeperConnection Watcher:127.0.0.1:63231 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 464982 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 464982 T1857 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 464984 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 464985 T1879 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@274f4c8c
> name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 464986 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 464987 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
>    [junit4]   2> 464990 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
>    [junit4]   2> 464993 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
>    [junit4]   2> 464995 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
>    [junit4]   2> 464998 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
>    [junit4]   2> 465001 T1857 oascc.SolrZkClient.makePath makePath:
> /live_nodes
>    [junit4]   2> 465002 T1857 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63235_
>    [junit4]   2> 465003 T1857 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:63235_
>    [junit4]   2> 465005 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
>    [junit4]   2> 465007 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
>    [junit4]   2> 465009 T1857 oasc.Overseer.close Overseer (id=null)
> closing
>    [junit4]   2> 465011 T1857
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:63235_
>    [junit4]   2> 465011 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
>    [junit4]   2> 465013 T1857 oasc.Overseer.start Overseer
> (id=92571507416563715-127.0.0.1:63235_-n_0000000000) starting
>    [junit4]   2> 465015 T1857 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
>    [junit4]   2> 465023 T1857
> oasc.OverseerAutoReplicaFailoverThread.<init> Starting
> OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000
> autoReplicaFailoverWaitAfterExpiration=30000
> autoReplicaFailoverBadNodeExpiration=60000
>    [junit4]   2> 465024 T1881 oasc.OverseerCollectionProcessor.run Process
> current queue of collection creations
>    [junit4]   2> 465024 T1857 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
>    [junit4]   2> 465026 T1857 oascc.SolrZkClient.makePath makePath:
> /aliases.json
>    [junit4]   2> 465028 T1857
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 465031 T1880 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
>    [junit4]   2> 465037 T1883 oasc.ZkController.publish publishing
> core=collection1 state=down collection=control_collection
>    [junit4]   2> 465037 T1883 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 465038 T1883 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 465039 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 465041 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1
> message={
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "numShards":"1",
>    [junit4]   2>          "base_url":"http://127.0.0.1:63235",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "node_name":"127.0.0.1:63235_",
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "collection":"control_collection",
>    [junit4]   2>          "core":"collection1"}
>    [junit4]   2> 465042 T1880
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> control_collection with shards [shard1]
>    [junit4]   2> 465042 T1880
> oasc.Overseer$ClusterStateUpdater.createCollection state version
> control_collection 1
>    [junit4]   2> 465042 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
>    [junit4]   2> 465045 T1884 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 466039 T1883 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 466040 T1883 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:control_collection
>    [junit4]   2> 466041 T1883 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 466041 T1883 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/control_collection
>    [junit4]   2> 466042 T1883 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
>    [junit4]   2> 466042 T1883 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/'
>    [junit4]   2> 466044 T1883 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/lib/README'
> to classloader
>    [junit4]   2> 466045 T1883 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 466046 T1883 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 466111 T1883 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 466157 T1883 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 466159 T1883 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 466170 T1883 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 466519 T1883 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 466522 T1883 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 466524 T1883 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 466529 T1883 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 466553 T1883 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection control_collection
>    [junit4]   2> 466554 T1883 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 466554 T1883 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/,
> dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data/
>    [junit4]   2> 466554 T1883 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@3d39f2c8
>    [junit4]   2> 466555 T1883 oasc.CachingDirectoryFactory.get return new
> directory for
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data
>    [junit4]   2> 466555 T1883 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data/index/
>    [junit4]   2> 466556 T1883 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 466556 T1883 oasc.CachingDirectoryFactory.get return new
> directory for
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data/index
>    [junit4]   2> 466556 T1883 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.MockRandomMergePolicy:
> org.apache.lucene.index.MockRandomMergePolicy@2fcbb1a4
>    [junit4]   2> 466557 T1883 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-8a18421)),segFN=segments_1,generation=1}
>    [junit4]   2> 466557 T1883 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 466562 T1883 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 466563 T1883 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 466563 T1883 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 466564 T1883 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 466564 T1883 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 466564 T1883 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 466565 T1883 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 466565 T1883 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 466565 T1883 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 466566 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 466566 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 466567 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 466567 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 466568 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 466568 T1883 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 466568 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 466569 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 466569 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 466570 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 466570 T1883 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 466589 T1883 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 466591 T1883 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 466593 T1883 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 466595 T1883 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 466600 T1883 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 466601 T1883 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 466601 T1883 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
> minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 466602 T1883 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-8a18421)),segFN=segments_1,generation=1}
>    [junit4]   2> 466602 T1883 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 466603 T1883 oass.SolrIndexSearcher.<init> Opening
> Searcher@6efb0cc[collection1] main
>    [junit4]   2> 466603 T1883 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/control_collection
>    [junit4]   2> 466604 T1883 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
>    [junit4]   2> 466604 T1883 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 466605 T1883
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 466605 T1883 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 466605 T1883 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 466606 T1883
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data
> found for znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 466606 T1883 oasr.ManagedResourceStorage.load Loaded null
> at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 466607 T1883 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/_rest_managed.json
>    [junit4]   2> 466609 T1883
> oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to
> new znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 466609 T1883
> oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 466609 T1883 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 466613 T1885 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@6efb0cc[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 466616 T1883 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 466617 T1888 oasc.ZkController.register Register replica
> - core:collection1 address:http://127.0.0.1:63235
> collection:control_collection shard:shard1
>    [junit4]   2> 466618 T1857 oass.SolrDispatchFilter.init
> user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
>    [junit4]   2> 466618 T1857 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 466619 T1888 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leader_elect/shard1/election
>    [junit4]   2> 466619 T1857
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 466620 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 466622 T1890 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1bb47d0f
> name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 466622 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 466622 T1857 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 466624 T1857
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 466625 T1888
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard1
>    [junit4]   2> 466627 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 466628 T1888
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
>    [junit4]   2> 466628 T1857 oasc.ChaosMonkey.monkeyLog monkey: init -
> expire sessions:false cause connection loss:false
>    [junit4]   2> 466628 T1888
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>    [junit4]   2> 466628 T1857
> oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with
> stateFormat=2
>    [junit4]   2> ASYNC  NEW_CORE C106 name=collection1
> org.apache.solr.core.SolrCore@17d8dfe6 url=
> http://127.0.0.1:63235/collection1 node=127.0.0.1:63235_
> C106_STATE=coll:control_collection core:collection1 props:{state=down,
> base_url=http://127.0.0.1:63235, node_name=127.0.0.1:63235_,
> core=collection1}
>    [junit4]   2> 466629 T1888 C106 P63235 oasc.SyncStrategy.sync Sync
> replicas to http://127.0.0.1:63235/collection1/
>    [junit4]   2> 466629 T1857
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 466629 T1888 C106 P63235 oasc.SyncStrategy.syncReplicas
> Sync Success - now sync replicas to me
>    [junit4]   2> 466630 T1888 C106 P63235 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:63235/collection1/ has no replicas
>    [junit4]   2> 466630 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 466630 T1888
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:63235/collection1/ shard1
>    [junit4]   2> 466631 T1888 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leaders/shard1
>    [junit4]   2> 466631 T1892 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@12f80eeb
> name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 466632 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 466632 T1857 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 466634 T1884 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 466634 T1893 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 466642 T1880
> oasc.Overseer$ClusterStateUpdater.buildCollection building a new
> collection: collection1
>    [junit4]   2> 466642 T1880
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> collection1 with shards [shard1, shard2, shard3]
>    [junit4]   2> 466642 T1880
> oasc.Overseer$ClusterStateUpdater.createCollection state version
> collection1 2
>    [junit4]   2> 466644 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 466645 T1880
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection
> /collections/collection1/state.json
>    [junit4]   2> 466646 T1893 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 466646 T1884 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 466654 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 466655 T1893 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 466655 T1884 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 466689 T1888 oasc.ZkController.register We are
> http://127.0.0.1:63235/collection1/ and leader is
> http://127.0.0.1:63235/collection1/
>    [junit4]   2> 466690 T1888 oasc.ZkController.register No LogReplay
> needed for core=collection1 baseURL=http://127.0.0.1:63235
>    [junit4]   2> 466690 T1888 oasc.ZkController.checkRecovery I am the
> leader, no recovery necessary
>    [junit4]   2> 466690 T1888 oasc.ZkController.publish publishing
> core=collection1 state=active collection=control_collection
>    [junit4]   2> 466690 T1888 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 466692 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 466696 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "state":"active",
>    [junit4]   2>          "numShards":"3",
>    [junit4]   2>          "base_url":"http://127.0.0.1:63235",
>    [junit4]   2>          "shard":"shard1",
>    [junit4]   2>          "node_name":"127.0.0.1:63235_",
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "core_node_name":"core_node1",
>    [junit4]   2>          "collection":"control_collection",
>    [junit4]   2>          "core":"collection1"}
>    [junit4]   2> 466801 T1884 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 466801 T1893 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 467932 T1857
> oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
>    [junit4]   2> 467933 T1857 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 467936 T1857 oejs.AbstractConnector.doStart Started
> SocketConnector@127.0.0.1:63242
>    [junit4]   2> 467936 T1857 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 467937 T1857 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 467937 T1857 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003
>    [junit4]   2> 467937 T1857 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/'
>    [junit4]   2> 467967 T1857 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/solr.xml
>    [junit4]   2> 468023 T1857 oasc.CoreContainer.<init> New CoreContainer
> 2090946628
>    [junit4]   2> 468023 T1857 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/]
>    [junit4]   2> 468024 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 468024 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 468024 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 468024 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 468025 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 468025 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 468025 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 468025 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 468026 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 468026 T1857 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 468026 T1857 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 468027 T1857 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 468027 T1857 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 468027 T1857 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:63231/solr
>    [junit4]   2> 468027 T1857 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 468028 T1857
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 468028 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 468030 T1904 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@429ce048
> name:ZooKeeperConnection Watcher:127.0.0.1:63231 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 468030 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 468031 T1857 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 468032 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 468033 T1906 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@4309255d
> name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 468034 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 468038 T1857
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 469043 T1857 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63242_
>    [junit4]   2> 469044 T1857 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:63242_
>    [junit4]   2> 469048 T1857 oasc.Overseer.close Overseer (id=null)
> closing
>    [junit4]   2> 469057 T1908 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 469057 T1908 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 469059 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 469060 T1908 oasc.ZkController.preRegister Registering
> watch for external collection collection1
>    [junit4]   2> 469060 T1908 oascc.ZkStateReader.addZkWatch addZkWatch
> collection1
>    [junit4]   2> 469060 T1908 oascc.ZkStateReader.addZkWatch Updating
> collection state at /collections/collection1/state.json from ZooKeeper...
>    [junit4]   2> 469062 T1908 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 0
>    [junit4]   2> 469062 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "numShards":"3",
>    [junit4]   2>          "base_url":"http://127.0.0.1:63242",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "node_name":"127.0.0.1:63242_",
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "core":"collection1"}
>    [junit4]   2> 469062 T1908 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 469065 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
>    [junit4]   2> 469066 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard2
>    [junit4]   2> 469071 T1880
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection
> /collections/collection1/state.json
>    [junit4]   2> 469072 T1907 oascc.ZkStateReader$7.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 469073 T1907 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 1
>    [junit4]   2> 470063 T1908 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 470064 T1908 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:collection1
>    [junit4]   2> 470065 T1908 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 470065 T1908 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 470066 T1908 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 470066 T1908 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/'
>    [junit4]   2> 470068 T1908 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 470068 T1908 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/lib/README'
> to classloader
>    [junit4]   2> 470069 T1908 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 470149 T1908 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 470195 T1908 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 470197 T1908 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 470207 T1908 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 470551 T1908 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 470553 T1908 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 470555 T1908 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 470560 T1908 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 470583 T1908 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 470583 T1908 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 470583 T1908 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/,
> dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1/
>    [junit4]   2> 470584 T1908 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@3d39f2c8
>    [junit4]   2> 470585 T1908 oasc.CachingDirectoryFactory.get return new
> directory for
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1
>    [junit4]   2> 470585 T1908 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1/index/
>    [junit4]   2> 470585 T1908 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 470586 T1908 oasc.CachingDirectoryFactory.get return new
> directory for
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1/index
>    [junit4]   2> 470586 T1908 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.MockRandomMergePolicy:
> org.apache.lucene.index.MockRandomMergePolicy@719085c4
>    [junit4]   2> 470587 T1908 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-28690035)),segFN=segments_1,generation=1}
>    [junit4]   2> 470588 T1908 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 470594 T1908 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 470595 T1908 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 470595 T1908 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 470595 T1908 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 470595 T1908 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 470596 T1908 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 470596 T1908 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 470597 T1908 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 470597 T1908 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 470598 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 470598 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 470599 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 470600 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 470600 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 470600 T1908 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 470601 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 470601 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 470602 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 470602 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 470603 T1908 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 470622 T1908 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 470624 T1908 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 470626 T1908 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 470627 T1908 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 470632 T1908 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 470632 T1908 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 470633 T1908 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
> minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 470634 T1908 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-28690035)),segFN=segments_1,generation=1}
>    [junit4]   2> 470634 T1908 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 470634 T1908 oass.SolrIndexSearcher.<init> Opening
> Searcher@5dbb42a8[collection1] main
>    [junit4]   2> 470634 T1908 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 470636 T1908 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 470636 T1908 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 470636 T1908
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 470636 T1908 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 470637 T1908 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 470638 T1908
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38
> bytes from znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 470638 T1908 oasr.ManagedResourceStorage.load Loaded
> LinkedHashMap at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 470638 T1908 oasr.ManagedResource.reloadFromStorage
> Loaded initArgs {} for /rest/managed
>    [junit4]   2> 470638 T1908 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 470643 T1909 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@5dbb42a8[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 470648 T1908 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 470649 T1912 oasc.ZkController.register Register replica
> - core:collection1 address:http://127.0.0.1:63242 collection:collection1
> shard:shard2
>    [junit4]   2> 470649 T1857 oass.SolrDispatchFilter.init
> user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
>    [junit4]   2> 470650 T1857 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 470650 T1912 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard2/election
>    [junit4]   2> 470656 T1912
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard2
>    [junit4]   2> 470657 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 470658 T1912
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
>    [junit4]   2> 470658 T1912
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>    [junit4]   2> ASYNC  NEW_CORE C107 name=collection1
> org.apache.solr.core.SolrCore@5d54e2ed url=
> http://127.0.0.1:63242/collection1 node=127.0.0.1:63242_
> C107_STATE=coll:collection1 core:collection1 props:{state=down, base_url=
> http://127.0.0.1:63242, node_name=127.0.0.1:63242_, core=collection1}
>    [junit4]   2> 470658 T1912 C107 P63242 oasc.SyncStrategy.sync Sync
> replicas to http://127.0.0.1:63242/collection1/
>    [junit4]   2> 470659 T1912 C107 P63242 oasc.SyncStrategy.syncReplicas
> Sync Success - now sync replicas to me
>    [junit4]   2> 470659 T1912 C107 P63242 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:63242/collection1/ has no replicas
>    [junit4]   2> 470659 T1912
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:63242/collection1/ shard2
>    [junit4]   2> 470659 T1912 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard2
>    [junit4]   2> 470664 T1880
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection
> /collections/collection1/state.json
>    [junit4]   2> 470665 T1907 oascc.ZkStateReader$7.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 470665 T1907 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 2
>    [junit4]   2> 470672 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 470672 T1880
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection
> /collections/collection1/state.json
>    [junit4]   2> 470673 T1907 oascc.ZkStateReader$7.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 470674 T1907 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 3
>    [junit4]   2> 470715 T1912 oasc.ZkController.register We are
> http://127.0.0.1:63242/collection1/ and leader is
> http://127.0.0.1:63242/collection1/
>    [junit4]   2> 470716 T1912 oasc.ZkController.register No LogReplay
> needed for core=collection1 baseURL=http://127.0.0.1:63242
>    [junit4]   2> 470716 T1912 oasc.ZkController.checkRecovery I am the
> leader, no recovery necessary
>    [junit4]   2> 470716 T1912 oasc.ZkController.publish publishing
> core=collection1 state=active collection=collection1
>    [junit4]   2> 470716 T1912 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 470718 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 470722 T1912 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 3
>    [junit4]   2> 470722 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "state":"active",
>    [junit4]   2>          "numShards":"3",
>    [junit4]   2>          "base_url":"http://127.0.0.1:63242",
>    [junit4]   2>          "shard":"shard2",
>    [junit4]   2>          "node_name":"127.0.0.1:63242_",
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "core_node_name":"core_node1",
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "core":"collection1"}
>    [junit4]   2> 470729 T1880
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection
> /collections/collection1/state.json
>    [junit4]   2> 470730 T1907 oascc.ZkStateReader$7.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 470730 T1907 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 4
>    [junit4]   2> 471948 T1857
> oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
>    [junit4]   2> 471949 T1857 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 471952 T1857 oejs.AbstractConnector.doStart Started
> SocketConnector@127.0.0.1:63247
>    [junit4]   2> 471952 T1857 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 471952 T1857 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 471952 T1857 oasc.SolrResourceLoader.locateSolrHome using
> system property solr.solr.home:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004
>    [junit4]   2> 471953 T1857 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/'
>    [junit4]   2> 471983 T1857 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/solr.xml
>    [junit4]   2> 472036 T1857 oasc.CoreContainer.<init> New CoreContainer
> 1701083293
>    [junit4]   2> 472037 T1857 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/]
>    [junit4]   2> 472038 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 472038 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 472038 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 472038 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 472040 T1857 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 472040 T1857 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 472040 T1857 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 472041 T1857 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 472041 T1857 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:63231/solr
>    [junit4]   2> 472041 T1857 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 472041 T1857
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 472042 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 472044 T1923 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@272abf6c
> name:ZooKeeperConnection Watcher:127.0.0.1:63231 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 472045 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 472045 T1857 oascc.SolrZkClient.createZkACLProvider Using
> default ZkACLProvider
>    [junit4]   2> 472047 T1857 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 472048 T1925 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@4f9e2851
> name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 472048 T1857 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 472054 T1857
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 473060 T1857 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63247_
>    [junit4]   2> 473061 T1857 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:63247_
>    [junit4]   2> 473065 T1857 oasc.Overseer.close Overseer (id=null)
> closing
>    [junit4]   2> 473075 T1927 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 473075 T1927 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 473077 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 473078 T1927 oasc.ZkController.preRegister Registering
> watch for external collection collection1
>    [junit4]   2> 473078 T1927 oascc.ZkStateReader.addZkWatch addZkWatch
> collection1
>    [junit4]   2> 473079 T1927 oascc.ZkStateReader.addZkWatch Updating
> collection state at /collections/collection1/state.json from ZooKeeper...
>    [junit4]   2> 473080 T1927 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 4
>    [junit4]   2> 473080 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3
> message={
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "numShards":"3",
>    [junit4]   2>          "base_url":"http://127.0.0.1:63247",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "node_name":"127.0.0.1:63247_",
>    [junit4]   2>          "operation":"state",
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "core":"collection1"}
>    [junit4]   2> 473081 T1927 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 473084 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=3
>    [junit4]   2> 473085 T1880
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
>    [junit4]   2> 473091 T1880
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection
> /collections/collection1/state.json
>    [junit4]   2> 473092 T1907 oascc.ZkStateReader$7.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 473092 T1926 oascc.ZkStateReader$7.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 473093 T1907 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 5
>    [junit4]   2> 473093 T1926 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 5
>    [junit4]   2> 474081 T1927 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 474082 T1927 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:collection1
>    [junit4]   2> 474083 T1927 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 474083 T1927 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 474084 T1927 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 474084 T1927 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/'
>    [junit4]   2> 474086 T1927 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 474086 T1927 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/lib/README'
> to classloader
>    [junit4]   2> 474087 T1927 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 474152 T1927 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 474198 T1927 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 474200 T1927 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 474211 T1927 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 474565 T1927 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 474568 T1927 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 474570 T1927 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 474575 T1927 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 474599 T1927 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 474600 T1927 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 474600 T1927 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/,
> dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2/
>    [junit4]   2> 474600 T1927 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@3d39f2c8
>    [junit4]   2> 474601 T1927 oasc.CachingDirectoryFactory.get return new
> directory for
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2
>    [junit4]   2> 474602 T1927 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2/index/
>    [junit4]   2> 474602 T1927 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 474603 T1927 oasc.CachingDirectoryFactory.get return new
> directory for
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2/index
>    [junit4]   2> 474603 T1927 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.MockRandomMergePolicy:
> org.apache.lucene.index.MockRandomMergePolicy@543ee3ab
>    [junit4]   2> 474604 T1927 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-469fc815)),segFN=segments_1,generation=1}
>    [junit4]   2> 474604 T1927 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 474611 T1927 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 474611 T1927 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 474612 T1927 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 474612 T1927 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 474612 T1927 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 474613 T1927 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 474613 T1927 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 474613 T1927 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 474614 T1927 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 474614 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 474615 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 474616 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 474616 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 474617 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 474617 T1927 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 474618 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 474618 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 474619 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 474619 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 474620 T1927 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 474639 T1927 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 474641 T1927 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 474643 T1927 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 474645 T1927 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 474649 T1927 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 474649 T1927 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 474650 T1927 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class
> org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy:
> minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 474651 T1927 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-469fc815)),segFN=segments_1,generation=1}
>    [junit4]   2> 474651 T1927 oasc.SolrDeletionPolicy.updateCommits newest
> commit generation = 1
>    [junit4]   2> 474652 T1927 oass.SolrIndexSearcher.<init> Opening
> Searcher@7e703b98[collection1] main
>    [junit4]   2> 474652 T1927 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 474653 T1927 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 474653 T1927 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 474654 T1927
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 474654 T1927 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 474654 T1927 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 474655 T1927
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38
> bytes from znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 474655 T1927 oasr.ManagedResourceStorage.load Loaded
> LinkedHashMap at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 474656 T1927 oasr.ManagedResource.reloadFromStorage
> Loaded initArgs {} for /rest/managed
>    [junit4]   2> 474656 T1927 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 474660 T1928 oasc.SolrCore.registerSearcher [collection1]
> Registered new searcher Searcher@7e703b98[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 474665 T1927 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 474666 T1931 oasc.ZkController.register Register replica
> - core:collection1 address:http://127.0.0.1:63247 collection:collection1
> shard:shard1
>    [junit4]   2> 474667 T1857 oass.SolrDispatchFilter.init
> user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
>    [junit4]   2> 474667 T1857 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 474667 T1931 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
>    [junit4]   2> 474672 T1931
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard1
>    [junit4]   2> 474674 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 474675 T1931
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
>    [junit4]   2> 474675 T1931
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>    [junit4]   2> ASYNC  NEW_CORE C108 name=collection1
> org.apache.solr.core.SolrCore@7d6e5880 url=
> http://127.0.0.1:63247/collection1 node=127.0.0.1:63247_
> C108_STATE=coll:collection1 core:collection1 props:{state=down, base_url=
> http://127.0.0.1:63247, node_name=127.0.0.1:63247_, core=collection1}
>    [junit4]   2> 474675 T1931 C108 P63247 oasc.SyncStrategy.sync Sync
> replicas to http://127.0.0.1:63247/collection1/
>    [junit4]   2> 474676 T1931 C108 P63247 oasc.SyncStrategy.syncReplicas
> Sync Success - now sync replicas to me
>    [junit4]   2> 474676 T1931 C108 P63247 oasc.SyncStrategy.syncToMe
> http://127.0.0.1:63247/collection1/ has no replicas
>    [junit4]   2> 474676 T1931
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> http://127.0.0.1:63247/collection1/ shard1
>    [junit4]   2> 474676 T1931 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard1
>    [junit4]   2> 474682 T1880
> oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection
> /collections/collection1/state.json
>    [junit4]   2> 474683 T1926 oascc.ZkStateReader$7.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 474683 T1907 oascc.ZkStateReader$7.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/collections/collection1/state.json, has occurred - updating...
>    [junit4]   2> 474684 T1926 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 6
>    [junit4]   2> 474684 T1907 oascc.ZkStateReader.updateWatchedCollection
> Updating data for collection1 to ver 6
>    [junit4]   2> 474691 T1884
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 474692 T1880
> oasc.Overseer$ClusterStateUpdater.updateZkStates goin
>
> [...truncated too long message...]
>
> e.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
>    [junit4]    >        at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable
> #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=46283, name=Thread-5354,
> state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
>    [junit4]    > Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could
> not fully create collection: awholynewstresscollection_collection1_2
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
>    [junit4]    >        at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable
> #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=46287, name=Thread-5358,
> state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
>    [junit4]    > Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could
> not fully create collection: awholynewstresscollection_collection5_2
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
>    [junit4]    >        at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable
> #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=46286, name=Thread-5357,
> state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
>    [junit4]    > Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could
> not fully create collection: awholynewstresscollection_collection4_2
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
>    [junit4]    >        at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable
> #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=46282, name=Thread-5353,
> state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
>    [junit4]    > Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could
> not fully create collection: awholynewstresscollection_collection0_2
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
>    [junit4]    >        at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable
> #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an
> uncaught exception in thread: Thread[id=46284, name=Thread-5355,
> state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
>    [junit4]    > Caused by:
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could
> not fully create collection: awholynewstresscollection_collection2_2
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
>    [junit4]    >        at
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
>    [junit4]    >        at
> org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
>    [junit4]    >        at
> org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
>    [junit4]   2> 1231178 T45904 oas.SolrTestCaseJ4.deleteCore ###deleteCore
>    [junit4]   2> NOTE: leaving temporary files on disk at:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-CB2CCA54206DAEAF-001
>    [junit4]   2> 885572 T45903 ccr.ThreadLeakControl.checkThreadLeaks
> WARNING Will linger awaiting termination of 1 leaked thread(s).
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50):
> {timestamp=FSTOrd41, text=OrdsLucene41(blocksize=128), a_t=FSTOrd41,
> intDefault=PostingsFormat(name=Lucene41VarGapFixedInterval),
> id=PostingsFormat(name=MockRandom), multiDefault=FSTOrd41, a_i1=FSTOrd41,
> _version_=PostingsFormat(name=Lucene41VarGapFixedInterval),
> range_facet_l=FSTOrd41,
> other_tl1=PostingsFormat(name=Lucene41VarGapFixedInterval)},
> docValues:{timestamp=DocValuesFormat(name=Asserting)},
> sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=es_NI,
> timezone=Africa/Sao_Tome
>    [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation
> 1.7.0_65 (64-bit)/cpus=16,threads=1,free=124971632,total=506462208
>    [junit4]   2> NOTE: All tests run in this JVM: [TestSearchPerf,
> TestReRankQParserPlugin, SearchHandlerTest, ResponseLogComponentTest,
> SOLR749Test, SpellPossibilityIteratorTest, TestCoreDiscovery,
> TestCopyFieldCollectionResource, DistributedIntervalFacetingTest,
> OverriddenZkACLAndCredentialsProvidersTest,
> DistributedQueryComponentOptimizationTest,
> DocExpirationUpdateProcessorFactoryTest, DistributedExpandComponentTest,
> StatelessScriptUpdateProcessorFactoryTest, HdfsSyncSliceTest,
> FileBasedSpellCheckerTest, CoreMergeIndexesAdminHandlerTest,
> TestSchemaNameResource, TestCSVLoader, SuggesterTest,
> TestFileDictionaryLookup, SpellCheckComponentTest,
> CoreContainerCoreInitFailuresTest, DocumentAnalysisRequestHandlerTest,
> CollectionsAPIDistributedZkTest]
>    [junit4] Completed on J3 in 886.12s, 1 test, 1 error <<< FAILURES!
>
> [...truncated 891 lines...]
> BUILD FAILED
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531:
> The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472:
> The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61:
> The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39:
> The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189:
> The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496:
> The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359:
> The following error occurred while executing this line:
> /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961:
> There were test failures: 436 suites, 1783 tests, 2 errors, 54 ignored (10
> assumptions)
>
> Total time: 207 minutes 22 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
> Archived 5 artifacts
> Archive block size is 32768
> Received 0 blocks and 684121688 bytes
> Compression is 0.0%
> Took 1 min 56 sec
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>



-- 
Regards,
Shalin Shekhar Mangar.

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 648 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/648/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
IOException occured when talking to server at: http://127.0.0.1:63235/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:63235/collection1
	at __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF:4ACA444C5732CE93]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:584)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:414)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:144)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:159)
	at org.apache.http.impl.io.AbstractSessionOutputBuffer.writeLine(AbstractSessionOutputBuffer.java:258)
	at org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:59)
	at org.apache.http.impl.io.HttpRequestWriter.writeHeadLine(HttpRequestWriter.java:46)
	at org.apache.http.impl.io.AbstractMessageWriter.write(AbstractMessageWriter.java:109)
	at org.apache.http.impl.AbstractHttpClientConnection.sendRequestHeader(AbstractHttpClientConnection.java:254)
	at org.apache.http.impl.conn.DefaultClientConnection.sendRequestHeader(DefaultClientConnection.java:270)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestHeader(ManagedClientConnectionImpl.java:209)
	at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:203)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:466)
	... 49 more




Build Log:
[...truncated 11952 lines...]
   [junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/init-core-data-001
   [junit4]   2> 463604 T1857 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 463613 T1857 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 463614 T1857 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 463614 T1858 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 463714 T1857 oasc.ZkTestServer.run start zk server on port:63231
   [junit4]   2> 463715 T1857 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 463716 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 463718 T1864 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ab7a30a name:ZooKeeperConnection Watcher:127.0.0.1:63231 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 463719 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 463719 T1857 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 463719 T1857 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 463721 T1857 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 463722 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 463723 T1866 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@653818d4 name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 463723 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 463723 T1857 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 463724 T1857 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 463726 T1857 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 463727 T1857 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 463729 T1857 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 463731 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 463731 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 463734 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 463734 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 463736 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 463737 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 463738 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 463739 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 463741 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 463741 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 463743 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 463743 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 463745 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 463746 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 463747 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 463748 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 463750 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 463750 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 463752 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 463752 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 463754 T1857 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 463755 T1857 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 464885 T1857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 464889 T1857 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63235
   [junit4]   2> 464889 T1857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 464890 T1857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 464890 T1857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002
   [junit4]   2> 464890 T1857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/'
   [junit4]   2> 464920 T1857 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/solr.xml
   [junit4]   2> 464974 T1857 oasc.CoreContainer.<init> New CoreContainer 239935073
   [junit4]   2> 464974 T1857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/]
   [junit4]   2> 464975 T1857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 464975 T1857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 464975 T1857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 464976 T1857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 464977 T1857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 464977 T1857 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 464977 T1857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 464978 T1857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 464978 T1857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 464978 T1857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63231/solr
   [junit4]   2> 464978 T1857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 464979 T1857 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 464979 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 464981 T1877 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6905799c name:ZooKeeperConnection Watcher:127.0.0.1:63231 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 464982 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 464982 T1857 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 464984 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 464985 T1879 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@274f4c8c name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 464986 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 464987 T1857 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 464990 T1857 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 464993 T1857 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 464995 T1857 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 464998 T1857 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 465001 T1857 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 465002 T1857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63235_
   [junit4]   2> 465003 T1857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63235_
   [junit4]   2> 465005 T1857 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 465007 T1857 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 465009 T1857 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 465011 T1857 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:63235_
   [junit4]   2> 465011 T1857 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 465013 T1857 oasc.Overseer.start Overseer (id=92571507416563715-127.0.0.1:63235_-n_0000000000) starting
   [junit4]   2> 465015 T1857 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 465023 T1857 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 465024 T1881 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 465024 T1857 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 465026 T1857 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 465028 T1857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 465031 T1880 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 465037 T1883 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 465037 T1883 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 465038 T1883 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 465039 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 465041 T1880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63235",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63235_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 465042 T1880 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 465042 T1880 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 465042 T1880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 465045 T1884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 466039 T1883 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 466040 T1883 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 466041 T1883 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 466041 T1883 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 466042 T1883 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 466042 T1883 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/'
   [junit4]   2> 466044 T1883 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 466045 T1883 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 466046 T1883 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 466111 T1883 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 466157 T1883 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 466159 T1883 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 466170 T1883 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 466519 T1883 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 466522 T1883 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 466524 T1883 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 466529 T1883 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 466553 T1883 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 466554 T1883 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 466554 T1883 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data/
   [junit4]   2> 466554 T1883 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d39f2c8
   [junit4]   2> 466555 T1883 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data
   [junit4]   2> 466555 T1883 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data/index/
   [junit4]   2> 466556 T1883 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 466556 T1883 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/control/data/index
   [junit4]   2> 466556 T1883 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2fcbb1a4
   [junit4]   2> 466557 T1883 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-8a18421)),segFN=segments_1,generation=1}
   [junit4]   2> 466557 T1883 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 466562 T1883 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 466563 T1883 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 466563 T1883 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 466564 T1883 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 466564 T1883 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 466564 T1883 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 466565 T1883 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 466565 T1883 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 466565 T1883 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 466566 T1883 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 466566 T1883 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 466567 T1883 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 466567 T1883 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 466568 T1883 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 466568 T1883 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 466568 T1883 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 466569 T1883 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 466569 T1883 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 466570 T1883 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 466570 T1883 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 466589 T1883 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 466591 T1883 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 466593 T1883 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 466595 T1883 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 466600 T1883 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 466601 T1883 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 466601 T1883 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 466602 T1883 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-8a18421)),segFN=segments_1,generation=1}
   [junit4]   2> 466602 T1883 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 466603 T1883 oass.SolrIndexSearcher.<init> Opening Searcher@6efb0cc[collection1] main
   [junit4]   2> 466603 T1883 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 466604 T1883 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 466604 T1883 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 466605 T1883 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 466605 T1883 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 466605 T1883 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 466606 T1883 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 466606 T1883 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 466607 T1883 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 466609 T1883 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 466609 T1883 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 466609 T1883 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 466613 T1885 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6efb0cc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 466616 T1883 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 466617 T1888 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63235 collection:control_collection shard:shard1
   [junit4]   2> 466618 T1857 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 466618 T1857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 466619 T1888 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 466619 T1857 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 466620 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 466622 T1890 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1bb47d0f name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 466622 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 466622 T1857 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 466624 T1857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 466625 T1888 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 466627 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466628 T1888 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 466628 T1857 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 466628 T1888 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 466628 T1857 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> ASYNC  NEW_CORE C106 name=collection1 org.apache.solr.core.SolrCore@17d8dfe6 url=http://127.0.0.1:63235/collection1 node=127.0.0.1:63235_ C106_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:63235, node_name=127.0.0.1:63235_, core=collection1}
   [junit4]   2> 466629 T1888 C106 P63235 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63235/collection1/
   [junit4]   2> 466629 T1857 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 466629 T1888 C106 P63235 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 466630 T1888 C106 P63235 oasc.SyncStrategy.syncToMe http://127.0.0.1:63235/collection1/ has no replicas
   [junit4]   2> 466630 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 466630 T1888 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63235/collection1/ shard1
   [junit4]   2> 466631 T1888 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 466631 T1892 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12f80eeb name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 466632 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 466632 T1857 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 466634 T1884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 466634 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 466642 T1880 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 466642 T1880 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 466642 T1880 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 466644 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466645 T1880 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 466646 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 466646 T1884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 466654 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466655 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 466655 T1884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 466689 T1888 oasc.ZkController.register We are http://127.0.0.1:63235/collection1/ and leader is http://127.0.0.1:63235/collection1/
   [junit4]   2> 466690 T1888 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63235
   [junit4]   2> 466690 T1888 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 466690 T1888 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 466690 T1888 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 466692 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 466696 T1880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63235",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "node_name":"127.0.0.1:63235_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 466801 T1884 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 466801 T1893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 467932 T1857 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 467933 T1857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 467936 T1857 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63242
   [junit4]   2> 467936 T1857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 467937 T1857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 467937 T1857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003
   [junit4]   2> 467937 T1857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/'
   [junit4]   2> 467967 T1857 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/solr.xml
   [junit4]   2> 468023 T1857 oasc.CoreContainer.<init> New CoreContainer 2090946628
   [junit4]   2> 468023 T1857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/]
   [junit4]   2> 468024 T1857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 468024 T1857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 468024 T1857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 468024 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 468025 T1857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 468025 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 468025 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 468025 T1857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 468026 T1857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 468026 T1857 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 468026 T1857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 468027 T1857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 468027 T1857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 468027 T1857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63231/solr
   [junit4]   2> 468027 T1857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 468028 T1857 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 468028 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 468030 T1904 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@429ce048 name:ZooKeeperConnection Watcher:127.0.0.1:63231 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 468030 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 468031 T1857 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 468032 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 468033 T1906 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4309255d name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 468034 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 468038 T1857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 469043 T1857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63242_
   [junit4]   2> 469044 T1857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63242_
   [junit4]   2> 469048 T1857 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 469057 T1908 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 469057 T1908 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 469059 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 469060 T1908 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 469060 T1908 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 469060 T1908 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 469062 T1908 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 469062 T1880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63242",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63242_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 469062 T1908 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 469065 T1880 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 469066 T1880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 469071 T1880 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 469072 T1907 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 469073 T1907 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 470063 T1908 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 470064 T1908 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 470065 T1908 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 470065 T1908 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 470066 T1908 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 470066 T1908 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/'
   [junit4]   2> 470068 T1908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 470068 T1908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 470069 T1908 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 470149 T1908 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 470195 T1908 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 470197 T1908 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 470207 T1908 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 470551 T1908 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 470553 T1908 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 470555 T1908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 470560 T1908 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 470583 T1908 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 470583 T1908 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 470583 T1908 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1/
   [junit4]   2> 470584 T1908 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d39f2c8
   [junit4]   2> 470585 T1908 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1
   [junit4]   2> 470585 T1908 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1/index/
   [junit4]   2> 470585 T1908 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 470586 T1908 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty1/index
   [junit4]   2> 470586 T1908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@719085c4
   [junit4]   2> 470587 T1908 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-28690035)),segFN=segments_1,generation=1}
   [junit4]   2> 470588 T1908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 470594 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 470595 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 470595 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 470595 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 470595 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 470596 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 470596 T1908 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 470597 T1908 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 470597 T1908 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 470598 T1908 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 470598 T1908 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 470599 T1908 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 470600 T1908 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 470600 T1908 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 470600 T1908 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 470601 T1908 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 470601 T1908 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 470602 T1908 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 470602 T1908 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 470603 T1908 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 470622 T1908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470624 T1908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470626 T1908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470627 T1908 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470632 T1908 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 470632 T1908 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 470633 T1908 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 470634 T1908 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-28690035)),segFN=segments_1,generation=1}
   [junit4]   2> 470634 T1908 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 470634 T1908 oass.SolrIndexSearcher.<init> Opening Searcher@5dbb42a8[collection1] main
   [junit4]   2> 470634 T1908 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 470636 T1908 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 470636 T1908 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 470636 T1908 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 470636 T1908 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 470637 T1908 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 470638 T1908 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 470638 T1908 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 470638 T1908 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 470638 T1908 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 470643 T1909 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5dbb42a8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 470648 T1908 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 470649 T1912 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63242 collection:collection1 shard:shard2
   [junit4]   2> 470649 T1857 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 470650 T1857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 470650 T1912 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 470656 T1912 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 470657 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470658 T1912 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 470658 T1912 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C107 name=collection1 org.apache.solr.core.SolrCore@5d54e2ed url=http://127.0.0.1:63242/collection1 node=127.0.0.1:63242_ C107_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:63242, node_name=127.0.0.1:63242_, core=collection1}
   [junit4]   2> 470658 T1912 C107 P63242 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63242/collection1/
   [junit4]   2> 470659 T1912 C107 P63242 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 470659 T1912 C107 P63242 oasc.SyncStrategy.syncToMe http://127.0.0.1:63242/collection1/ has no replicas
   [junit4]   2> 470659 T1912 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63242/collection1/ shard2
   [junit4]   2> 470659 T1912 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 470664 T1880 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 470665 T1907 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 470665 T1907 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 470672 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470672 T1880 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 470673 T1907 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 470674 T1907 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 470715 T1912 oasc.ZkController.register We are http://127.0.0.1:63242/collection1/ and leader is http://127.0.0.1:63242/collection1/
   [junit4]   2> 470716 T1912 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:63242
   [junit4]   2> 470716 T1912 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 470716 T1912 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 470716 T1912 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 470718 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 470722 T1912 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 470722 T1880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63242",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "node_name":"127.0.0.1:63242_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 470729 T1880 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 470730 T1907 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 470730 T1907 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 471948 T1857 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 471949 T1857 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 471952 T1857 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:63247
   [junit4]   2> 471952 T1857 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 471952 T1857 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 471952 T1857 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004
   [junit4]   2> 471953 T1857 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/'
   [junit4]   2> 471983 T1857 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/solr.xml
   [junit4]   2> 472036 T1857 oasc.CoreContainer.<init> New CoreContainer 1701083293
   [junit4]   2> 472037 T1857 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/]
   [junit4]   2> 472038 T1857 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 472038 T1857 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 472038 T1857 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 472038 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 472039 T1857 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 472040 T1857 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 472040 T1857 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 472040 T1857 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 472041 T1857 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 472041 T1857 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:63231/solr
   [junit4]   2> 472041 T1857 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 472041 T1857 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 472042 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 472044 T1923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@272abf6c name:ZooKeeperConnection Watcher:127.0.0.1:63231 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 472045 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 472045 T1857 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 472047 T1857 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 472048 T1925 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f9e2851 name:ZooKeeperConnection Watcher:127.0.0.1:63231/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 472048 T1857 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 472054 T1857 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 473060 T1857 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63247_
   [junit4]   2> 473061 T1857 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:63247_
   [junit4]   2> 473065 T1857 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 473075 T1927 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 473075 T1927 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 473077 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 473078 T1927 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 473078 T1927 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 473079 T1927 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 473080 T1927 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 473080 T1880 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:63247",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:63247_",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 473081 T1927 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 473084 T1880 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 473085 T1880 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 473091 T1880 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 473092 T1907 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 473092 T1926 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 473093 T1907 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 473093 T1926 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 474081 T1927 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 474082 T1927 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 474083 T1927 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 474083 T1927 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 474084 T1927 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 474084 T1927 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/'
   [junit4]   2> 474086 T1927 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 474086 T1927 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 474087 T1927 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 474152 T1927 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 474198 T1927 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 474200 T1927 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 474211 T1927 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 474565 T1927 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 474568 T1927 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 474570 T1927 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 474575 T1927 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 474599 T1927 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 474600 T1927 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 474600 T1927 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2/
   [junit4]   2> 474600 T1927 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3d39f2c8
   [junit4]   2> 474601 T1927 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2
   [junit4]   2> 474602 T1927 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2/index/
   [junit4]   2> 474602 T1927 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 474603 T1927 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest-CB2CCA54206DAEAF-001/tempDir-001/jetty2/index
   [junit4]   2> 474603 T1927 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@543ee3ab
   [junit4]   2> 474604 T1927 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-469fc815)),segFN=segments_1,generation=1}
   [junit4]   2> 474604 T1927 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 474611 T1927 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 474611 T1927 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 474612 T1927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 474612 T1927 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 474612 T1927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 474613 T1927 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 474613 T1927 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 474613 T1927 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 474614 T1927 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 474614 T1927 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 474615 T1927 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 474616 T1927 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 474616 T1927 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 474617 T1927 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 474617 T1927 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 474618 T1927 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 474618 T1927 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 474619 T1927 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 474619 T1927 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 474620 T1927 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 474639 T1927 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 474641 T1927 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 474643 T1927 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 474645 T1927 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 474649 T1927 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 474649 T1927 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 474650 T1927 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=35, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 474651 T1927 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=BaseDirectoryWrapper(RAMDirectory(id=lucene-469fc815)),segFN=segments_1,generation=1}
   [junit4]   2> 474651 T1927 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 474652 T1927 oass.SolrIndexSearcher.<init> Opening Searcher@7e703b98[collection1] main
   [junit4]   2> 474652 T1927 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 474653 T1927 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 474653 T1927 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 474654 T1927 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 474654 T1927 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 474654 T1927 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 474655 T1927 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 474655 T1927 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 474656 T1927 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 474656 T1927 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 474660 T1928 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e703b98[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 474665 T1927 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 474666 T1931 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:63247 collection:collection1 shard:shard1
   [junit4]   2> 474667 T1857 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 474667 T1857 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 474667 T1931 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 474672 T1931 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 474674 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474675 T1931 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 474675 T1931 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C108 name=collection1 org.apache.solr.core.SolrCore@7d6e5880 url=http://127.0.0.1:63247/collection1 node=127.0.0.1:63247_ C108_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:63247, node_name=127.0.0.1:63247_, core=collection1}
   [junit4]   2> 474675 T1931 C108 P63247 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:63247/collection1/
   [junit4]   2> 474676 T1931 C108 P63247 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 474676 T1931 C108 P63247 oasc.SyncStrategy.syncToMe http://127.0.0.1:63247/collection1/ has no replicas
   [junit4]   2> 474676 T1931 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:63247/collection1/ shard1
   [junit4]   2> 474676 T1931 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 474682 T1880 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 474683 T1926 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 474683 T1907 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 474684 T1926 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 474684 T1907 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 474691 T1884 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 474692 T1880 oasc.Overseer$ClusterStateUpdater.updateZkStates goin

[...truncated too long message...]

e.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=46283, name=Thread-5354, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection1_2
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=46287, name=Thread-5358, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection5_2
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=46286, name=Thread-5357, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection4_2
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=46282, name=Thread-5353, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection0_2
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=46284, name=Thread-5355, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection2_2
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([CB2CCA54206DAEAF]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1635)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1661)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]   2> 1231178 T45904 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-CB2CCA54206DAEAF-001
   [junit4]   2> 885572 T45903 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {timestamp=FSTOrd41, text=OrdsLucene41(blocksize=128), a_t=FSTOrd41, intDefault=PostingsFormat(name=Lucene41VarGapFixedInterval), id=PostingsFormat(name=MockRandom), multiDefault=FSTOrd41, a_i1=FSTOrd41, _version_=PostingsFormat(name=Lucene41VarGapFixedInterval), range_facet_l=FSTOrd41, other_tl1=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=es_NI, timezone=Africa/Sao_Tome
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=124971632,total=506462208
   [junit4]   2> NOTE: All tests run in this JVM: [TestSearchPerf, TestReRankQParserPlugin, SearchHandlerTest, ResponseLogComponentTest, SOLR749Test, SpellPossibilityIteratorTest, TestCoreDiscovery, TestCopyFieldCollectionResource, DistributedIntervalFacetingTest, OverriddenZkACLAndCredentialsProvidersTest, DistributedQueryComponentOptimizationTest, DocExpirationUpdateProcessorFactoryTest, DistributedExpandComponentTest, StatelessScriptUpdateProcessorFactoryTest, HdfsSyncSliceTest, FileBasedSpellCheckerTest, CoreMergeIndexesAdminHandlerTest, TestSchemaNameResource, TestCSVLoader, SuggesterTest, TestFileDictionaryLookup, SpellCheckComponentTest, CoreContainerCoreInitFailuresTest, DocumentAnalysisRequestHandlerTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J3 in 886.12s, 1 test, 1 error <<< FAILURES!

[...truncated 891 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 436 suites, 1783 tests, 2 errors, 54 ignored (10 assumptions)

Total time: 207 minutes 22 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 684121688 bytes
Compression is 0.0%
Took 1 min 56 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 647 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/647/

1 tests failed.
FAILED:  org.apache.solr.client.solrj.TestLBHttpSolrServer.testReliability

Error Message:
No live SolrServers available to handle this request

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
	at __randomizedtesting.SeedInfo.seed([8FE0961F3C8F3FF8:4E284B599DE9EE51]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:528)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.client.solrj.TestLBHttpSolrServer.testReliability(TestLBHttpSolrServer.java:223)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:55703/solr
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:580)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:481)
	... 43 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:466)
	... 46 more




Build Log:
[...truncated 13392 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.TestLBHttpSolrServer
   [junit4]   2> SLF4J: Class path contains multiple SLF4J bindings.
   [junit4]   2> SLF4J: Found binding in [jar:file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/solrj/test-lib/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   [junit4]   2> SLF4J: Found binding in [jar:file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/lib/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
   [junit4]   2> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
   [junit4]   2> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
   [junit4]   2> log4j:WARN No such property [conversionPattern] in org.apache.solr.util.SolrLogLayout.
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/init-core-data-001
   [junit4]   2> 4265 T11 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 4464 T11 oas.SolrTestCaseJ4.setUp ###Starting testReliability
   [junit4]   2> 6020 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 6517 T11 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:19643
   [junit4]   2> 6732 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 6910 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 6911 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001
   [junit4]   2> 6912 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/'
   [junit4]   2> 8193 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/solr.xml
   [junit4]   2> 8524 T11 oasc.CoreContainer.<init> New CoreContainer 2068971112
   [junit4]   2> 8534 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/]
   [junit4]   2> 8826 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 8827 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 8828 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 8829 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 8830 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 8833 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 8833 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 8834 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 8835 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 9295 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 9325 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 9334 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 9348 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 10027 T23 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/'
   [junit4]   2> 10289 T23 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 10789 T23 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 10853 T23 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 10909 T23 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 11344 T23 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 11374 T23 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/
   [junit4]   2> 11806 T23 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 11890 T23 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/
   [junit4]   2> 11891 T23 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 12160 T23 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data
   [junit4]   2> 12163 T23 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index/
   [junit4]   2> 12165 T23 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 12201 T23 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index
   [junit4]   2> 12596 T23 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 12604 T23 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 14561 T23 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 14687 T23 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14688 T23 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14689 T23 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14777 T23 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 14778 T23 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 14779 T23 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 14788 T23 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 14824 T23 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 14927 T23 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 15158 T23 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 15197 T23 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 15200 T23 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 15203 T23 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 15361 T23 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 15361 T23 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 15389 T23 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 15390 T23 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 15569 T23 oass.SolrIndexSearcher.<init> Opening Searcher@20b8610d[collection1] main
   [junit4]   2> 15630 T23 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf
   [junit4]   2> 15631 T23 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf}
   [junit4]   2> 15775 T23 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf
   [junit4]   2> 15777 T23 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 15794 T23 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf
   [junit4]   2> 15795 T23 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 15987 T23 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 15988 T23 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 16000 T24 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20b8610d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 16008 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1
   [junit4]   2> 16009 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C0 name=collection1 org.apache.solr.core.SolrCore@989c09e
   [junit4]   2> 18947 T21 C0 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} 0 1139
   [junit4]   2> 19242 T21 C0 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> ASYNC  NEW_CORE C1 name=collection1 org.apache.solr.core.SolrCore@989c09e
   [junit4]   2> 20472 T21 C1 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index),segFN=segments_2,generation=2}
   [junit4]   2> 20474 T21 C1 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 20867 T21 C1 oass.SolrIndexSearcher.<init> Opening Searcher@3181b4a2[collection1] main
   [junit4]   2> 20879 T21 C1 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 20900 T24 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3181b4a2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c10)))}
   [junit4]   2> 20902 T21 C1 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={commit=true&softCommit=false&version=2&wt=javabin&waitSearcher=true} {commit=} 0 1660
   [junit4]   2> 20933 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 20937 T11 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:45054
   [junit4]   2> 20939 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 20940 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 20942 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001
   [junit4]   2> 20943 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/'
   [junit4]   2> 21026 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/solr.xml
   [junit4]   2> 21182 T11 oasc.CoreContainer.<init> New CoreContainer 1873517704
   [junit4]   2> 21183 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/]
   [junit4]   2> 21185 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 21186 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 21187 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 21188 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 21189 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 21189 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 21190 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 21191 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 21192 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 21193 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 21194 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 21195 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 21196 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 21204 T35 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/'
   [junit4]   2> 21264 T35 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 21341 T35 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 21343 T35 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 21354 T35 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 21379 T35 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 21380 T35 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/
   [junit4]   2> 21381 T35 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 21382 T35 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/
   [junit4]   2> 21383 T35 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 21386 T35 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data
   [junit4]   2> 21388 T35 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index/
   [junit4]   2> 21389 T35 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 21392 T35 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index
   [junit4]   2> 21400 T35 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 21401 T35 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 21403 T35 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 21404 T35 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21405 T35 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21406 T35 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21406 T35 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 21407 T35 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 21407 T35 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 21408 T35 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 21408 T35 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 21409 T35 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 21434 T35 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21437 T35 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21439 T35 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21441 T35 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21444 T35 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 21445 T35 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 21449 T35 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 21449 T35 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 21450 T35 oass.SolrIndexSearcher.<init> Opening Searcher@246d79a7[collection1] main
   [junit4]   2> 21451 T35 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/conf
   [junit4]   2> 21451 T35 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/conf}
   [junit4]   2> 21452 T35 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/conf
   [junit4]   2> 21454 T35 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 21455 T35 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/conf
   [junit4]   2> 21455 T35 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 21456 T35 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 21456 T36 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@246d79a7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 21456 T35 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 21458 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1
   [junit4]   2> 21458 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@74a27fae
   [junit4]   2> 21469 T28 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} 0 7
   [junit4]   2> 21473 T28 C2 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 21512 T28 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index),segFN=segments_2,generation=2}
   [junit4]   2> 21513 T28 C2 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 21518 T28 C2 oass.SolrIndexSearcher.<init> Opening Searcher@4ddc7d76[collection1] main
   [junit4]   2> 21519 T28 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 21519 T36 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ddc7d76[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c10)))}
   [junit4]   2> 21520 T28 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={commit=true&softCommit=false&version=2&wt=javabin&waitSearcher=true} {commit=} 0 47
   [junit4]   2> 21533 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 21535 T11 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:55703
   [junit4]   2> 21537 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 21537 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 21538 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001
   [junit4]   2> 21538 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/'
   [junit4]   2> 21556 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/solr.xml
   [junit4]   2> 21629 T11 oasc.CoreContainer.<init> New CoreContainer 268728179
   [junit4]   2> 21630 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/]
   [junit4]   2> 21632 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 21632 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 21633 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 21633 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 21634 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 21634 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 21635 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 21635 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 21647 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 21648 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 21649 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 21649 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 21650 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 21656 T47 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/'
   [junit4]   2> 21711 T47 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 21764 T47 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 21765 T47 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 21775 T47 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 21791 T47 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 21792 T47 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/
   [junit4]   2> 21793 T47 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 21794 T47 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/
   [junit4]   2> 21794 T47 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 21796 T47 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data
   [junit4]   2> 21798 T47 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index/
   [junit4]   2> 21799 T47 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 21801 T47 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index
   [junit4]   2> 21808 T47 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 21808 T47 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 21811 T47 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 21812 T47 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21812 T47 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21813 T47 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 21813 T47 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 21814 T47 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 21814 T47 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 21814 T47 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 21815 T47 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 21815 T47 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 21818 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21820 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21822 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21824 T47 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 21827 T47 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 21827 T47 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 21831 T47 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 21831 T47 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 21832 T47 oass.SolrIndexSearcher.<init> Opening Searcher@49f09ea6[collection1] main
   [junit4]   2> 21833 T47 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/conf
   [junit4]   2> 21833 T47 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/conf}
   [junit4]   2> 21834 T47 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/conf
   [junit4]   2> 21835 T47 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 21836 T47 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/conf
   [junit4]   2> 21836 T47 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 21837 T47 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 21838 T48 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@49f09ea6[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 21838 T47 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 21839 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1
   [junit4]   2> 21840 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@616001af
   [junit4]   2> 21850 T40 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {add=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]} 0 7
   [junit4]   2> 21854 T40 C3 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 21891 T40 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index),segFN=segments_2,generation=2}
   [junit4]   2> 21892 T40 C3 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 21896 T40 C3 oass.SolrIndexSearcher.<init> Opening Searcher@13e915b5[collection1] main
   [junit4]   2> 21897 T40 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 21897 T48 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13e915b5[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c10)))}
   [junit4]   2> 21898 T40 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={commit=true&softCommit=false&version=2&wt=javabin&waitSearcher=true} {commit=} 0 44
   [junit4]   2> 21901 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1873517704
   [junit4]   2> 21902 T11 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@74a27fae
   [junit4]   2> 21903 T11 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 21904 T11 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 21905 T11 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 21905 T11 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 21908 T11 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 21909 T11 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 21910 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index;done=false>>]
   [junit4]   2> 21911 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data/index
   [junit4]   2> 22204 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data;done=false>>]
   [junit4]   2> 22205 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-1-001/collection1/data
   [junit4]   2> 22209 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 27761 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2068971112
   [junit4]   2> 27761 T50 oahic.DefaultRequestDirector.tryExecute I/O exception (java.net.SocketException) caught when processing request: Connection reset
   [junit4]   2> 27763 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 27764 T50 oahic.DefaultRequestDirector.tryExecute Retrying request
   [junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@989c09e
   [junit4]   2> 27889 T17 C4 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={version=2&wt=javabin&q=*:*} hits=10 status=0 QTime=903 
   [junit4]   2> ASYNC  NEW_CORE C5 name=collection1 org.apache.solr.core.SolrCore@616001af
   [junit4]   2> 27889 T46 C5 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={version=2&wt=javabin&q=*:*} hits=10 status=0 QTime=379 
   [junit4]   2> 27980 T17 C4 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:207)
   [junit4]   2> 		at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:98)
   [junit4]   2> 		at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
   [junit4]   2> 		at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:50)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:771)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:424)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 27982 T17 C4 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:207)
   [junit4]   2> 		at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:98)
   [junit4]   2> 		at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
   [junit4]   2> 		at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:50)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:771)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:424)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 28051 T19 C4 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={sort=_docid_+asc&version=2&rows=0&wt=javabin&distrib=false&q=*:*} hits=10 status=0 QTime=291 
   [junit4]   2> 28053 T19 C4 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:207)
   [junit4]   2> 		at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:98)
   [junit4]   2> 		at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
   [junit4]   2> 		at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:50)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:771)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:424)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 28055 T19 C4 oasc.SolrException.log ERROR null:org.eclipse.jetty.io.EofException
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:142)
   [junit4]   2> 		at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:214)
   [junit4]   2> 		at org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:207)
   [junit4]   2> 		at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:98)
   [junit4]   2> 		at org.apache.solr.response.BinaryResponseWriter.write(BinaryResponseWriter.java:52)
   [junit4]   2> 		at org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:50)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:771)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:424)
   [junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:202)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:137)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
   [junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
   [junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
   [junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
   [junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:368)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
   [junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
   [junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
   [junit4]   2> 		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
   [junit4]   2> 		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 		at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	
   [junit4]   2> 28058 T19 C4 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@989c09e
   [junit4]   2> 28059 T19 C4 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 28060 T19 C4 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 28061 T19 C4 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 28061 T19 C4 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 28070 T19 C4 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 28072 T19 C4 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 28072 T19 C4 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index;done=false>>]
   [junit4]   2> 28073 T19 C4 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index
   [junit4]   2> 28093 T19 C4 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data;done=false>>]
   [junit4]   2> 28094 T19 C4 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data
   [junit4]   2> 28274 T44 C5 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={sort=_docid_+asc&version=2&rows=0&wt=javabin&distrib=false&q=*:*} hits=10 status=0 QTime=1 
   [junit4]   2> 32799 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=268728179
   [junit4]   2> 32801 T11 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@616001af
   [junit4]   2> 32801 T11 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 32803 T11 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 32803 T11 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 32804 T11 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 32811 T11 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 32813 T11 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 32814 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data;done=false>>]
   [junit4]   2> 32814 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data
   [junit4]   2> 32816 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index;done=false>>]
   [junit4]   2> 32817 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index
   [junit4]   2> 32836 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 37809 T11 oas.SolrTestCaseJ4.tearDown ###Ending testReliability
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLBHttpSolrServer -Dtests.method=testReliability -Dtests.seed=8FE0961F3C8F3FF8 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ro_RO -Dtests.timezone=Atlantic/Canary -Dtests.file.encoding=UTF-8
   [junit4] ERROR   33.6s J1 | TestLBHttpSolrServer.testReliability <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8FE0961F3C8F3FF8:4E284B599DE9EE51]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:528)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.client.solrj.TestLBHttpSolrServer.testReliability(TestLBHttpSolrServer.java:223)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:55703/solr
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:580)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:481)
   [junit4]    > 	... 43 more
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:466)
   [junit4]    > 	... 46 more
   [junit4]   2> 38049 T11 oas.SolrTestCaseJ4.setUp ###Starting testSimple
   [junit4]   2> 38061 T11 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 38064 T11 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:64378
   [junit4]   2> 38066 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 38067 T11 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 38067 T11 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001
   [junit4]   2> 38068 T11 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/'
   [junit4]   2> 38119 T11 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/solr.xml
   [junit4]   2> 38232 T11 oasc.CoreContainer.<init> New CoreContainer 1430551838
   [junit4]   2> 38233 T11 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/]
   [junit4]   2> 38235 T11 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 38236 T11 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 38237 T11 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 38238 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 38238 T11 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 38239 T11 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 38240 T11 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 38240 T11 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 38241 T11 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 38242 T11 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 38243 T11 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 38244 T11 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 38245 T11 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 38252 T60 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/'
   [junit4]   2> 38323 T60 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 38385 T60 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 38386 T60 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 38398 T60 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 38422 T60 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 38424 T60 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/
   [junit4]   2> 38425 T60 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockFSDirectoryFactory
   [junit4]   2> 38426 T60 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/
   [junit4]   2> 38426 T60 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 38429 T60 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data
   [junit4]   2> 38431 T60 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index/
   [junit4]   2> 38432 T60 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 38434 T60 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index
   [junit4]   2> 38441 T60 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 38442 T60 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 38444 T60 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 38445 T60 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 38446 T60 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 38447 T60 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 38447 T60 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 38448 T60 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 38448 T60 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 38449 T60 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 38450 T60 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 38451 T60 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 38468 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38471 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38473 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38476 T60 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 38480 T60 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 38480 T60 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 38484 T60 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NIOFSDirectory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index lockFactory=NativeFSLockFactory@/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index),segFN=segments_1,generation=1}
   [junit4]   2> 38485 T60 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 38486 T60 oass.SolrIndexSearcher.<init> Opening Searcher@e9bb131[collection1] main
   [junit4]   2> 38487 T60 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf
   [junit4]   2> 38487 T60 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf}
   [junit4]   2> 38489 T60 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf
   [junit4]   2> 38490 T60 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 38491 T60 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf
   [junit4]   2> 38491 T60 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 38492 T60 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 38493 T61 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e9bb131[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 38493 T60 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 38494 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1
   [junit4]   2> 38495 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@461ea70e
   [junit4]   2> 38508 T53 C6 oasup.LogUpdateProcessor.finish [collection1] webapp=/solr path=/update params={version=2&wt=javabin} {ad

[...truncated too long message...]

lrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf
   [junit4]   2> 74627 T146 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using file:dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/conf
   [junit4]   2> 74627 T146 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 74628 T146 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 74628 T146 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 74629 T147 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7f34784e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c10)))}
   [junit4]   2> 74629 T146 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 74630 T11 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1
   [junit4]   2> 74631 T11 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C18 name=collection1 org.apache.solr.core.SolrCore@113d8793
   [junit4]   2> 74743 T139 C18 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={sort=_docid_+asc&version=2&rows=0&wt=javabin&distrib=false&q=*:*} hits=10 status=0 QTime=1 
   [junit4]   2> 75836 T139 C18 oasc.SolrCore.execute [collection1] webapp=/solr path=/select params={version=2&wt=javabin&q=*:*} hits=10 status=0 QTime=0 
   [junit4]   2> 75840 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1630454106
   [junit4]   2> 75841 T11 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@113d8793
   [junit4]   2> 75842 T11 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 75842 T11 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 75843 T11 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 75843 T11 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 75847 T11 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 75848 T11 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 75849 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data;done=false>>]
   [junit4]   2> 75849 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data
   [junit4]   2> 75850 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index;done=false>>]
   [junit4]   2> 75851 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-0-001/collection1/data/index
   [junit4]   2> 75865 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 80855 T11 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1636053566
   [junit4]   2> 80857 T11 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@181ce810
   [junit4]   2> 80857 T11 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=10,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 80858 T11 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 80859 T11 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 80859 T11 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 80863 T11 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 80864 T11 oasc.CachingDirectoryFactory.close Closing MockFSDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 80864 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data;done=false>>]
   [junit4]   2> 80865 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data
   [junit4]   2> 80867 T11 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index;done=false>>]
   [junit4]   2> 80867 T11 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001/instance-2-001/collection1/data/index
   [junit4]   2> 80882 T11 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 85864 T11 oas.SolrTestCaseJ4.tearDown ###Ending testTwoServers
   [junit4]   2> 85871 T11 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-solrj/test/J1/temp/solr.client.solrj.TestLBHttpSolrServer-8FE0961F3C8F3FF8-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {name=PostingsFormat(name=Lucene41VarGapFixedInterval), id=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=ro_RO, timezone=Atlantic/Canary
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=4,free=152531464,total=240123904
   [junit4]   2> NOTE: All tests run in this JVM: [TestLBHttpSolrServer]
   [junit4] Completed on J1 in 86.74s, 3 tests, 1 error <<< FAILURES!

[...truncated 96 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:202: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 53 suites, 314 tests, 1 error

Total time: 265 minutes 36 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 10748425 bytes
Compression is 0.0%
Took 2.8 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 646 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/646/

4 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
The Monkey ran for over 20 seconds and no jetties were stopped - this is worth investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 20 seconds and no jetties were stopped - this is worth investigating!
	at __randomizedtesting.SeedInfo.seed([9A887F063BF0EBD8:1B6EF11E4CAF8BE4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:535)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:140)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=284, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)    2) Thread[id=285, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 
   1) Thread[id=284, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
   2) Thread[id=285, name=StopableIndexingThread, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:121)
	at __randomizedtesting.SeedInfo.seed([9A887F063BF0EBD8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=284, name=StopableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.io.PrintStream.println(PrintStream.java:806)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:107)    2) Thread[id=285, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]         at java.io.FileOutputStream.writeBytes(Native Method)         at java.io.FileOutputStream.write(FileOutputStream.java:345)         at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)         at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)         at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:113)         at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.string(JsonWriter.java:535)         at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.value(JsonWriter.java:364)         at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:86)         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)         at java.io.PrintStream.flush(PrintStream.java:338)         at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)         at java.io.PrintStream.write(PrintStream.java:482)         at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)         at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)         at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)         at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)         at java.io.PrintStream.newLine(PrintStream.java:546)         at java.io.PrintStream.println(PrintStream.java:824)         at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)         at java.lang.Throwable.printStackTrace(Throwable.java:658)         at java.lang.Throwable.printStackTrace(Throwable.java:643)         at java.lang.Throwable.printStackTrace(Throwable.java:634)         at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:108)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=284, name=StopableIndexingThread, state=BLOCKED, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.io.PrintStream.println(PrintStream.java:806)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:107)
   2) Thread[id=285, name=StopableIndexingThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:345)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
        at com.carrotsearch.ant.tasks.junit4.slave.EventsOutputStream.write(EventsOutputStream.java:28)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:113)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:194)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.string(JsonWriter.java:535)
        at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.stream.JsonWriter.value(JsonWriter.java:364)
        at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:86)
        at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:410)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at java.io.PrintStream.flush(PrintStream.java:338)
        at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
        at java.io.PrintStream.write(PrintStream.java:482)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
        at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
        at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
        at java.io.PrintStream.newLine(PrintStream.java:546)
        at java.io.PrintStream.println(PrintStream.java:824)
        at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:748)
        at java.lang.Throwable.printStackTrace(Throwable.java:658)
        at java.lang.Throwable.printStackTrace(Throwable.java:643)
        at java.lang.Throwable.printStackTrace(Throwable.java:634)
        at org.apache.solr.cloud.StopableIndexingThread.run(StopableIndexingThread.java:108)
	at __randomizedtesting.SeedInfo.seed([9A887F063BF0EBD8]:0)


REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:21199, https://127.0.0.1:42998, https://127.0.0.1:58576, https://127.0.0.1:36320, https://127.0.0.1:32977]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:21199, https://127.0.0.1:42998, https://127.0.0.1:58576, https://127.0.0.1:36320, https://127.0.0.1:32977]
	at __randomizedtesting.SeedInfo.seed([9A887F063BF0EBD8:1B6EF11E4CAF8BE4]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:172)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:145)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard1/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 47 more




Build Log:
[...truncated 11413 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/init-core-data-001
   [junit4]   2> 114933 T188 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 114933 T188 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /jruk/
   [junit4]   2> 114951 T188 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 114989 T188 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 115000 T189 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 115302 T188 oasc.ZkTestServer.run start zk server on port:64683
   [junit4]   2> 115314 T188 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 115627 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 115800 T195 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2323368c name:ZooKeeperConnection Watcher:127.0.0.1:64683 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115801 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 115802 T188 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 115813 T188 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 115935 T188 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 115937 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 115939 T197 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6837725 name:ZooKeeperConnection Watcher:127.0.0.1:64683/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 115939 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 115940 T188 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 115948 T188 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 115953 T188 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 115957 T188 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 115961 T188 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 115965 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 115966 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 115973 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 115974 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 115979 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115980 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 115984 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 115985 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 115990 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 115990 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 115995 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 115996 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 116001 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 116002 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 116007 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 116007 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 116012 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 116013 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 116018 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 116018 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 116023 T188 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 116024 T188 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 118405 T188 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 118411 T188 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:64694
   [junit4]   2> 118411 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 118412 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 118413 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002
   [junit4]   2> 118413 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002/'
   [junit4]   2> 118478 T188 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002/solr.xml
   [junit4]   2> 118578 T188 oasc.CoreContainer.<init> New CoreContainer 430565162
   [junit4]   2> 118579 T188 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002/]
   [junit4]   2> 118580 T188 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 118581 T188 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 118581 T188 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 118582 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 118582 T188 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 118583 T188 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 118583 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 118583 T188 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 118584 T188 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 118585 T188 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 118585 T188 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 118586 T188 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 118586 T188 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 118587 T188 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64683/solr
   [junit4]   2> 118645 T188 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 118646 T188 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 118647 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 118651 T208 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2456538 name:ZooKeeperConnection Watcher:127.0.0.1:64683 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 118651 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 118652 T188 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 118690 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 118692 T210 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c0bc409 name:ZooKeeperConnection Watcher:127.0.0.1:64683/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 118692 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 118730 T188 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 118752 T188 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 118770 T188 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 118776 T188 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 118781 T188 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 118810 T188 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 118813 T188 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64694_jruk
   [junit4]   2> 118825 T188 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64694_jruk
   [junit4]   2> 118831 T188 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 118835 T188 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 118843 T188 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 118879 T188 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:64694_jruk
   [junit4]   2> 118880 T188 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 118884 T188 oasc.Overseer.start Overseer (id=92557117267968003-127.0.0.1:64694_jruk-n_0000000000) starting
   [junit4]   2> 118920 T188 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 118976 T188 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 119007 T212 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 119008 T188 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 119012 T188 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 119015 T188 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 119081 T211 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 119108 T214 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 119108 T214 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 119131 T214 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 119140 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 119180 T211 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64694/jruk",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64694_jruk",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null}
   [junit4]   2> 119181 T211 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 119220 T211 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 119243 T211 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 119269 T215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 120133 T214 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 120133 T214 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 120135 T214 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 120135 T214 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 120137 T214 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 120137 T214 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002/collection1/'
   [junit4]   2> 120140 T214 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 120141 T214 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 120142 T214 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 120308 T214 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 120377 T214 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 120379 T214 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 120391 T214 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 120869 T214 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 120873 T214 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 120875 T214 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 120887 T214 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 120947 T214 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 120948 T214 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 121030 T214 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/control/data/
   [junit4]   2> 121031 T214 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0a5970
   [junit4]   2> 121053 T214 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/control/data
   [junit4]   2> 121055 T214 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/control/data/index/
   [junit4]   2> 121056 T214 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 121057 T214 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/control/data/index
   [junit4]   2> 121057 T214 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 121078 T214 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 121078 T214 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 121090 T214 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 121202 T214 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 121203 T214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 121203 T214 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 121203 T214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 121204 T214 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 121204 T214 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 121205 T214 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 121205 T214 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 121214 T214 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 121215 T214 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 121216 T214 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 121218 T214 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 121219 T214 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 121219 T214 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 121220 T214 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 121221 T214 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 121222 T214 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 121223 T214 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 121224 T214 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 121246 T214 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 121248 T214 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 121250 T214 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 121252 T214 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 121268 T214 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 121268 T214 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 121270 T214 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4e033333
   [junit4]   2> 121272 T214 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 121272 T214 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 121273 T214 oass.SolrIndexSearcher.<init> Opening Searcher@57c39581[collection1] main
   [junit4]   2> 121273 T214 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 121275 T214 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 121276 T214 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 121288 T214 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 121288 T214 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 121289 T214 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 121290 T214 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 121291 T214 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 121292 T214 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 121296 T214 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 121297 T214 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 121297 T214 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 121307 T216 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@57c39581[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 121319 T214 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 121321 T219 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:64694/jruk collection:control_collection shard:shard1
   [junit4]   2> 121321 T188 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 121322 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 121360 T219 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 121370 T219 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 121374 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 121374 T219 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 121375 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 121385 T188 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> ASYNC  NEW_CORE C13 name=collection1 org.apache.solr.core.SolrCore@67618412 url=http://127.0.0.1:64694/jruk/collection1 node=127.0.0.1:64694_jruk C13_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:64694/jruk, node_name=127.0.0.1:64694_jruk, state=down}
   [junit4]   2> 121385 T219 C13 P64694 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:64694/jruk/collection1/
   [junit4]   2> 121386 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 121386 T219 C13 P64694 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 121387 T219 C13 P64694 oasc.SyncStrategy.syncToMe http://127.0.0.1:64694/jruk/collection1/ has no replicas
   [junit4]   2> 121387 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:64694/jruk/collection1/ shard1
   [junit4]   2> 121388 T221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67ed34f3 name:ZooKeeperConnection Watcher:127.0.0.1:64683/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 121388 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 121389 T188 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 121392 T215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 121392 T188 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 121401 T188 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 121401 T188 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 121402 T188 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 121403 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 121405 T223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d158b21 name:ZooKeeperConnection Watcher:127.0.0.1:64683/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 121405 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 121405 T188 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 121406 T219 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 121409 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 121417 T211 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 121417 T211 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 121418 T211 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 121423 T211 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 121426 T215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 121426 T224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 121451 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 121453 T224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 121453 T215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 121468 T219 oasc.ZkController.register We are http://127.0.0.1:64694/jruk/collection1/ and leader is http://127.0.0.1:64694/jruk/collection1/
   [junit4]   2> 121469 T219 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:64694/jruk
   [junit4]   2> 121469 T219 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 121469 T219 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 121470 T219 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 121472 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 121480 T211 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64694/jruk",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64694_jruk",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1"}
   [junit4]   2> 121587 T224 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 121587 T215 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 123492 T188 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 123493 T188 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 123497 T188 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:64703
   [junit4]   2> 123498 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 123498 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 123499 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003
   [junit4]   2> 123499 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003/'
   [junit4]   2> 123564 T188 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003/solr.xml
   [junit4]   2> 123675 T188 oasc.CoreContainer.<init> New CoreContainer 1563502472
   [junit4]   2> 123676 T188 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003/]
   [junit4]   2> 123678 T188 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 123678 T188 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 123678 T188 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 123679 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 123679 T188 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 123680 T188 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 123680 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 123681 T188 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 123681 T188 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 123682 T188 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 123683 T188 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 123683 T188 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 123684 T188 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 123684 T188 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64683/solr
   [junit4]   2> 123685 T188 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 123685 T188 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 123686 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 123688 T235 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@461a8184 name:ZooKeeperConnection Watcher:127.0.0.1:64683 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 123689 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 123690 T188 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 123693 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 123695 T237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6581a654 name:ZooKeeperConnection Watcher:127.0.0.1:64683/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 123695 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 123706 T188 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 124716 T188 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64703_jruk
   [junit4]   2> 124718 T188 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64703_jruk
   [junit4]   2> 124724 T188 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 124747 T239 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 124747 T239 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 124750 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 124752 T239 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 124753 T239 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 124753 T239 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 124756 T211 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64703/jruk",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64703_jruk",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null}
   [junit4]   2> 124762 T211 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 124763 T239 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 124763 T239 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 124764 T211 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 124775 T211 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 124777 T238 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 124778 T238 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 125764 T239 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 125765 T239 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 125766 T239 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 125767 T239 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 125768 T239 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 125769 T239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003/collection1/'
   [junit4]   2> 125771 T239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 125771 T239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 125773 T239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 125887 T239 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 125945 T239 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 125947 T239 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 125959 T239 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 126486 T239 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 126494 T239 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 126497 T239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 126503 T239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 126559 T239 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 126559 T239 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 126560 T239 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty1/
   [junit4]   2> 126560 T239 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0a5970
   [junit4]   2> 126567 T239 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty1
   [junit4]   2> 126568 T239 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty1/index/
   [junit4]   2> 126569 T239 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 126570 T239 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty1/index
   [junit4]   2> 126571 T239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 126591 T239 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 126592 T239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 126605 T239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 126605 T239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 126606 T239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 126606 T239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 126606 T239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 126607 T239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 126607 T239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 126608 T239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 126608 T239 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 126610 T239 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 126612 T239 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 126614 T239 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 126616 T239 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 126618 T239 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 126618 T239 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 126619 T239 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 126621 T239 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 126621 T239 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 126623 T239 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 126625 T239 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 126651 T239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 126654 T239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 126656 T239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 126658 T239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 126662 T239 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 126663 T239 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 126665 T239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6edc3dfb
   [junit4]   2> 126667 T239 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 126668 T239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 126668 T239 oass.SolrIndexSearcher.<init> Opening Searcher@6659893c[collection1] main
   [junit4]   2> 126669 T239 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 126671 T239 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 126671 T239 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 126672 T239 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 126672 T239 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 126673 T239 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 126675 T239 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 126675 T239 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 126675 T239 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 126676 T239 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 126691 T240 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6659893c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 126707 T239 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 126708 T243 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:64703/jruk collection:collection1 shard:shard3
   [junit4]   2> 126710 T188 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1
   [junit4]   2> 126711 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 126711 T243 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 126722 T243 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 126725 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 126726 T243 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 126726 T243 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C14 name=collection1 org.apache.solr.core.SolrCore@58014017 url=http://127.0.0.1:64703/jruk/collection1 node=127.0.0.1:64703_jruk C14_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:64703/jruk, node_name=127.0.0.1:64703_jruk, state=down}
   [junit4]   2> 126727 T243 C14 P64703 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:64703/jruk/collection1/
   [junit4]   2> 126727 T243 C14 P64703 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 126728 T243 C14 P64703 oasc.SyncStrategy.syncToMe http://127.0.0.1:64703/jruk/collection1/ has no replicas
   [junit4]   2> 126728 T243 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:64703/jruk/collection1/ shard3
   [junit4]   2> 126729 T243 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 126739 T211 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 126741 T238 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 126742 T238 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 126755 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 126757 T211 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 126758 T238 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 126760 T238 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 126791 T243 oasc.ZkController.register We are http://127.0.0.1:64703/jruk/collection1/ and leader is http://127.0.0.1:64703/jruk/collection1/
   [junit4]   2> 126792 T243 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:64703/jruk
   [junit4]   2> 126792 T243 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 126792 T243 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 126793 T243 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 126795 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 126801 T243 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 126801 T211 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64703/jruk",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64703_jruk",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard3"}
   [junit4]   2> 126813 T211 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 126814 T238 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 126815 T238 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 129060 T188 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 129062 T188 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 129064 T188 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:64713
   [junit4]   2> 129065 T188 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 129065 T188 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 129065 T188 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004
   [junit4]   2> 129066 T188 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004/'
   [junit4]   2> 129100 T188 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004/solr.xml
   [junit4]   2> 129188 T188 oasc.CoreContainer.<init> New CoreContainer 74752464
   [junit4]   2> 129189 T188 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004/]
   [junit4]   2> 129190 T188 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 129191 T188 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 129191 T188 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 129191 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 129191 T188 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 129192 T188 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 129192 T188 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 129192 T188 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 129193 T188 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 129193 T188 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 129194 T188 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 129194 T188 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 129194 T188 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 129195 T188 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:64683/solr
   [junit4]   2> 129195 T188 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 129195 T188 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 129196 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 129198 T254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52a2c76f name:ZooKeeperConnection Watcher:127.0.0.1:64683 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 129199 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 129199 T188 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 129202 T188 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 129203 T256 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5108c21a name:ZooKeeperConnection Watcher:127.0.0.1:64683/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 129203 T188 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 129210 T188 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 130218 T188 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64713_jruk
   [junit4]   2> 130219 T188 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64713_jruk
   [junit4]   2> 130226 T188 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 130241 T258 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 130241 T258 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 130244 T215 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 130246 T258 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 130246 T258 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 130247 T258 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 130250 T258 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 130250 T211 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:64713/jruk",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:64713_jruk",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null}
   [junit4]   2> 130250 T258 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 130257 T211 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 130259 T211 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 130270 T211 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 130271 T257 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 130271 T238 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 130272 T257 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 130273 T238 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 131251 T258 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 131252 T258 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 131253 T258 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 131254 T258 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 131255 T258 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 131255 T258 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004/collection1/'
   [junit4]   2> 131258 T258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 131259 T258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 131260 T258 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 131332 T258 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 131382 T258 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 131384 T258 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 131395 T258 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 131800 T258 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 131803 T258 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 131805 T258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 131812 T258 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 131865 T258 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 131865 T258 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 131865 T258 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty2/
   [junit4]   2> 131866 T258 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d0a5970
   [junit4]   2> 131873 T258 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty2
   [junit4]   2> 131874 T258 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty2/index/
   [junit4]   2> 131875 T258 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 131877 T258 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty2/index
   [junit4]   2> 131877 T258 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=29, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 131882 T258 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-9A887F063BF0EBD8-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 131883 T258 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 131896 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 131897 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 131897 T258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 131897 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 131898 T258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 131898 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 131899 T258 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 131899 T258 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 131899 T258 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 131901 T258 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 131903 T258 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 131904 T258 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 131906 T258 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 131908 T258 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 131908 T258 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 131909 T258 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 131910 T258 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 131911 T258 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 131913 T258 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 131914 T258 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 131936 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 131938 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 131941 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 131943 T258 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 131948 T258 oasu

[...truncated too long message...]

  2> 3166325 T7083 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 3166326 T7083 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 3166326 T7083 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 3166326 T7083 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 3166327 T7083 oasc.SolrCore.closeSearcher [delLiveColl_shard2_replica1] Closing main searcher on request.
   [junit4]   2> 3166327 T7083 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 3166328 T7083 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest-9A887F063BF0EBD8-001/tempDir-006/delLiveColl_shard2_replica1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest-9A887F063BF0EBD8-001/tempDir-006/delLiveColl_shard2_replica1/data/index;done=false>>]
   [junit4]   2> 3166328 T7083 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest-9A887F063BF0EBD8-001/tempDir-006/delLiveColl_shard2_replica1/data/index
   [junit4]   2> 3166329 T7083 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest-9A887F063BF0EBD8-001/tempDir-006/delLiveColl_shard2_replica1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest-9A887F063BF0EBD8-001/tempDir-006/delLiveColl_shard2_replica1/data;done=false>>]
   [junit4]   2> 3166329 T7083 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest-9A887F063BF0EBD8-001/tempDir-006/delLiveColl_shard2_replica1/data
   [junit4]   2> 3166330 T7083 oasc.Overseer.close Overseer (id=92557302840164366-127.0.0.1:42998_-n_0000000004) closing
   [junit4]   2> 3166330 T7254 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:42998_
   [junit4]   2> 3166333 T7253 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 3166334 T7083 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 3170665 T7083 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 3170667 T7083 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39305 39305
   [junit4]   2> 3170703 T7084 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39305 39305
   [junit4]   2> 3170704 T7084 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/aliases.json
   [junit4]   2> 		8	/solr/clusterstate.json
   [junit4]   2> 		4	/solr/collections/delLiveColl/state.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=9A887F063BF0EBD8 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=fr_BE -Dtests.timezone=Etc/GMT-7 -Dtests.file.encoding=UTF-8
   [junit4] ERROR    224s J2 | DeleteReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:21199, https://127.0.0.1:42998, https://127.0.0.1:58576, https://127.0.0.1:36320, https://127.0.0.1:32977]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9A887F063BF0EBD8:1B6EF11E4CAF8BE4]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:172)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:145)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard1/core_node1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 47 more
   [junit4]   2> 3170768 T7083 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.DeleteReplicaTest-9A887F063BF0EBD8-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=fr_BE, timezone=Etc/GMT-7
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=197259752,total=406323200
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrQueryParserResource, QueryResultKeyTest, NumericFieldsTest, TestDFRSimilarityFactory, TestInitQParser, PingRequestHandlerTest, TestQueryUtils, BasicFunctionalityTest, CollectionsAPIAsyncDistributedZkTest, TestSurroundQueryParser, TestManagedSchema, TestFieldTypeResource, ShardSplitTest, TestRandomDVFaceting, TestHashPartitioner, TestRealTimeGet, TestReplicationHandler, RegexBoostProcessorTest, QueryParsingTest, TestIndexingPerformance, UpdateRequestProcessorFactoryTest, SyncSliceTest, ZkNodePropsTest, CursorPagingTest, TestCloudInspectUtil, TestPerFieldSimilarity, TestBlendedInfixSuggestions, StatsComponentTest, PolyFieldTest, TestMissingGroups, SolrXmlInZkTest, TestFieldResource, HighlighterConfigTest, MultiTermTest, LeaderInitiatedRecoveryOnCommitTest, TestDocBasedVersionConstraints, TestStressLucene, QueryElevationComponentTest, CopyFieldTest, TestDistributedMissingSort, HardAutoCommitTest, TestDistributedSearch, SpellCheckCollatorTest, TestRemoteStreaming, DeleteReplicaTest]
   [junit4] Completed on J2 in 223.92s, 1 test, 1 error <<< FAILURES!

[...truncated 38 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 436 suites, 1783 tests, 2 suite-level errors, 1 error, 1 failure, 905 ignored (4 assumptions)

Total time: 178 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 165164552 bytes
Compression is 0.0%
Took 30 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 645 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/645/

2 tests failed.
REGRESSION:  org.apache.solr.SolrInfoMBeanTest.testCallMBeanInfo

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([29DDB92365BE42DE:D6BB341F0EC63FC0]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.junit.Assert.assertNotNull(Assert.java:537)
	at org.apache.solr.SolrInfoMBeanTest.testCallMBeanInfo(SolrInfoMBeanTest.java:66)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11618 lines...]
   [junit4] Suite: org.apache.solr.SolrInfoMBeanTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001
   [junit4]   2> 469642 T2431 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 469645 T2431 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 469645 T2431 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 469647 T2431 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 469648 T2431 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 469649 T2431 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 469864 T2431 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 469937 T2431 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 469938 T2431 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 469950 T2431 oass.IndexSchema.readSchema [null] Schema name=test
   [junit4]   2> 470227 T2431 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 470245 T2431 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 470247 T2431 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 470255 T2431 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 470259 T2431 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 470261 T2431 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 470262 T2431 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 470263 T2431 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 470263 T2431 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 470263 T2431 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 470263 T2431 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 470264 T2431 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 470264 T2431 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr
   [junit4]   2> 470264 T2431 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/'
   [junit4]   2> 470307 T2431 oasc.CoreContainer.<init> New CoreContainer 1296165388
   [junit4]   2> 470307 T2431 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/]
   [junit4]   2> 470308 T2431 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 0
   [junit4]   2> 470308 T2431 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 470308 T2431 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 0
   [junit4]   2> 470309 T2431 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 470309 T2431 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 470309 T2431 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 470309 T2431 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 470309 T2431 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 470309 T2431 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 470312 T2431 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=false
   [junit4]   2> 470313 T2431 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 470313 T2431 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 470313 T2431 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 470318 T2432 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 470319 T2432 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 470319 T2432 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 470319 T2432 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 470377 T2432 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 470441 T2432 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 470442 T2432 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 470453 T2432 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 470742 T2432 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 470748 T2432 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 470750 T2432 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 470758 T2432 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 470762 T2432 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 470765 T2432 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 470766 T2432 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 470766 T2432 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 470766 T2432 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 470767 T2432 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 470767 T2432 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 470767 T2432 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 470767 T2432 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 470768 T2432 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001/
   [junit4]   2> 470768 T2432 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@50d8a1bb
   [junit4]   2> 470768 T2432 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001
   [junit4]   2> 470769 T2432 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001/index/
   [junit4]   2> 470769 T2432 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 470769 T2432 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001/index
   [junit4]   2> 470769 T2432 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=76.955078125, floorSegmentMB=0.46484375, forceMergeDeletesPctAllowed=21.820705033847826, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 470770 T2432 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-40289de))),segFN=segments_1,generation=1}
   [junit4]   2> 470770 T2432 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 470770 T2432 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 470774 T2432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 470774 T2432 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 470774 T2432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 470774 T2432 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 470774 T2432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 470775 T2432 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 470775 T2432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 470775 T2432 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 470775 T2432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 470776 T2432 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 470776 T2432 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 470776 T2432 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 470776 T2432 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 470777 T2432 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 470777 T2432 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 470777 T2432 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 470777 T2432 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 470778 T2432 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 470778 T2432 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 470778 T2432 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 470779 T2432 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 470779 T2432 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 470779 T2432 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 470780 T2432 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 470780 T2432 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 470780 T2432 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 470780 T2432 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 470781 T2432 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 470781 T2432 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 470781 T2432 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 470782 T2432 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 470782 T2432 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 470782 T2432 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 470783 T2432 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 470783 T2432 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 470783 T2432 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 470783 T2432 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 470784 T2432 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 470807 T2432 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470809 T2432 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470810 T2432 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470812 T2432 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 470813 T2432 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 470816 T2432 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 470816 T2432 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 470816 T2432 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=54.8173828125, floorSegmentMB=0.8251953125, forceMergeDeletesPctAllowed=17.56012328655008, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 470817 T2432 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-40289de))),segFN=segments_1,generation=1}
   [junit4]   2> 470817 T2432 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 470817 T2432 oass.SolrIndexSearcher.<init> Opening Searcher@3f1eb085[collection1] main
   [junit4]   2> 470818 T2432 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 470818 T2432 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 470818 T2432 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 470818 T2432 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 470818 T2432 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 470818 T2432 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 470840 T2432 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 470894 T2433 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3f1eb085[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 470899 T2432 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 470901 T2431 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 470905 T2431 oas.SolrTestCaseJ4.setUp ###Starting testCallMBeanInfo
   [junit4]   2> 472433 T2431 oas.SolrTestCaseJ4.tearDown ###Ending testCallMBeanInfo
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SolrInfoMBeanTest -Dtests.method=testCallMBeanInfo -Dtests.seed=29DDB92365BE42DE -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=hu_HU -Dtests.timezone=Africa/Asmara -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 1.61s J3 | SolrInfoMBeanTest.testCallMBeanInfo <<<
   [junit4]    > Throwable #1: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([29DDB92365BE42DE:D6BB341F0EC63FC0]:0)
   [junit4]    > 	at org.apache.solr.SolrInfoMBeanTest.testCallMBeanInfo(SolrInfoMBeanTest.java:66)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 472516 T2431 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 472516 T2431 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1296165388
   [junit4]   2> 472517 T2431 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@460b39be
   [junit4]   2> 472534 T2431 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 472534 T2431 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 472534 T2431 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 472535 T2431 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 472537 T2431 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 472537 T2431 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 472537 T2431 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 472538 T2431 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001/index
   [junit4]   2> 472538 T2431 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001 [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001;done=false>>]
   [junit4]   2> 472538 T2431 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.SolrInfoMBeanTest-29DDB92365BE42DE-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=hu_HU, timezone=Africa/Asmara
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=60168288,total=357564416
   [junit4]   2> NOTE: All tests run in this JVM: [ClusterStateUpdateTest, TestSchemaVersionResource, DistributedTermsComponentTest, TestManagedSchemaDynamicFieldResource, TestSort, TestReplicationHandlerBackup, TestCursorMarkWithoutUniqueKey, TestPivotHelperCode, TestJoin, DocValuesMultiTest, TestSolrXmlPersistor, FieldMutatingUpdateProcessorTest, TestDefaultSearchFieldResource, BlockDirectoryTest, MigrateRouteKeyTest, TestSimpleQParserPlugin, UUIDFieldTest, TestCSVLoader, TermVectorComponentDistributedTest, DistributedSuggestComponentTest, TestOrdValues, TestSearchPerf, DocValuesMissingTest, SolrInfoMBeanTest]
   [junit4] Completed on J3 in 2.93s, 1 test, 1 failure <<< FAILURES!

[...truncated 407 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/init-core-data-001
   [junit4]   2> 994823 T5475 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 994823 T5475 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 994827 T5475 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 994828 T5475 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 994828 T5476 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 994928 T5475 oasc.ZkTestServer.run start zk server on port:46715
   [junit4]   2> 994929 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 994929 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 994932 T5482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e9989c7 name:ZooKeeperConnection Watcher:127.0.0.1:46715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 994933 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 994933 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 994933 T5475 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 994936 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 994936 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 994937 T5484 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78de159c name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 994937 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 994938 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 994938 T5475 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 994940 T5475 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 994941 T5475 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 994942 T5475 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 994943 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 994944 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 994946 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 994946 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 994948 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 994948 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 994950 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 994950 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 994952 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 994952 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 994953 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 994954 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 994955 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 994956 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 994957 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 994957 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 994959 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 994959 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 994961 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 994961 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 994963 T5475 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 994963 T5475 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 994965 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 994965 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 994966 T5486 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49b56a0f name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 994966 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 994966 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 996096 T5475 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 996099 T5475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 996101 T5475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:46723
   [junit4]   2> 996103 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 996103 T5475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 996103 T5475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-002
   [junit4]   2> 996104 T5475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-002/'
   [junit4]   2> 996133 T5475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-002/solr.xml
   [junit4]   2> 996204 T5475 oasc.CoreContainer.<init> New CoreContainer 402548169
   [junit4]   2> 996204 T5475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-002/]
   [junit4]   2> 996205 T5475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 996206 T5475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 996206 T5475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 996206 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 996206 T5475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 996206 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 996207 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 996207 T5475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 996207 T5475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 996209 T5475 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 996211 T5475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 996211 T5475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 996211 T5475 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 996211 T5475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46715/solr
   [junit4]   2> 996212 T5475 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 996212 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 996212 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 996214 T5497 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7633d6ed name:ZooKeeperConnection Watcher:127.0.0.1:46715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 996215 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 996215 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 996217 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 996218 T5499 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6549297d name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 996218 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 996219 T5475 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 996221 T5475 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 996222 T5475 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 996224 T5475 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 996226 T5475 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 996228 T5475 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 996229 T5475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46723_
   [junit4]   2> 996230 T5475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46723_
   [junit4]   2> 996231 T5475 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 996232 T5475 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 996234 T5475 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 996235 T5475 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:46723_
   [junit4]   2> 996235 T5475 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 996236 T5475 oasc.Overseer.start Overseer (id=92553004632899588-127.0.0.1:46723_-n_0000000000) starting
   [junit4]   2> 996238 T5475 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 996244 T5475 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 996244 T5501 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 996246 T5475 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 996247 T5475 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 996248 T5475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 996250 T5500 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 996252 T5475 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 996252 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 996255 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 996256 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 996257 T5505 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b9554f9 name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 996257 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 996257 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 996258 T5475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 996260 T5475 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 996260 T5475 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 996260 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 996261 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 996262 T5507 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30e9475e name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 996262 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 996262 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 996264 T5508 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 996266 T5500 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 996266 T5500 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 996266 T5500 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 996267 T5500 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 996268 T5508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 996268 T5509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 997262 T5475 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 997263 T5475 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 997267 T5475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 997269 T5475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:46730
   [junit4]   2> 997271 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 997271 T5475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 997272 T5475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-003
   [junit4]   2> 997272 T5475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-003/'
   [junit4]   2> 997314 T5475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-003/solr.xml
   [junit4]   2> 997378 T5475 oasc.CoreContainer.<init> New CoreContainer 978284643
   [junit4]   2> 997378 T5475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-003/]
   [junit4]   2> 997379 T5475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 997380 T5475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 997380 T5475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 997380 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 997380 T5475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 997380 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 997381 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 997381 T5475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 997381 T5475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 997383 T5475 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 997385 T5475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 997385 T5475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 997385 T5475 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 997385 T5475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46715/solr
   [junit4]   2> 997386 T5475 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 997386 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 997387 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 997388 T5520 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53b1b0fc name:ZooKeeperConnection Watcher:127.0.0.1:46715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 997389 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 997389 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 997391 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 997392 T5522 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@670dfb1a name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 997392 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 997396 T5475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 998400 T5475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46730_
   [junit4]   2> 998401 T5475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46730_
   [junit4]   2> 998405 T5475 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 998410 T5475 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 998410 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 999459 T5475 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 999460 T5475 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 999464 T5475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 999465 T5475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:46737
   [junit4]   2> 999468 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 999468 T5475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 999468 T5475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-004
   [junit4]   2> 999469 T5475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-004/'
   [junit4]   2> 999498 T5475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-004/solr.xml
   [junit4]   2> 999550 T5475 oasc.CoreContainer.<init> New CoreContainer 736460775
   [junit4]   2> 999550 T5475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-004/]
   [junit4]   2> 999551 T5475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 999551 T5475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 999552 T5475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 999552 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 999552 T5475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 999552 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 999553 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 999553 T5475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 999553 T5475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 999555 T5475 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 999557 T5475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 999557 T5475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 999557 T5475 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 999557 T5475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46715/solr
   [junit4]   2> 999558 T5475 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 999558 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 999558 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 999560 T5535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@47930d0a name:ZooKeeperConnection Watcher:127.0.0.1:46715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 999561 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 999561 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 999563 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 999564 T5537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27fe07f9 name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 999564 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 999568 T5475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1000572 T5475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46737_
   [junit4]   2> 1000573 T5475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46737_
   [junit4]   2> 1000577 T5475 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1000582 T5475 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 1000582 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1001923 T5475 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1001923 T5475 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1001927 T5475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1001929 T5475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:46747
   [junit4]   2> 1001931 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1001931 T5475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1001931 T5475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-005
   [junit4]   2> 1001932 T5475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-005/'
   [junit4]   2> 1001961 T5475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-005/solr.xml
   [junit4]   2> 1002013 T5475 oasc.CoreContainer.<init> New CoreContainer 1926580340
   [junit4]   2> 1002013 T5475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-005/]
   [junit4]   2> 1002045 T5475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1002046 T5475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1002046 T5475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1002046 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1002046 T5475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1002046 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1002047 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1002047 T5475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1002047 T5475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1002049 T5475 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1002051 T5475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1002051 T5475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1002051 T5475 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1002051 T5475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46715/solr
   [junit4]   2> 1002052 T5475 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1002052 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1002053 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1002054 T5550 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@407b4112 name:ZooKeeperConnection Watcher:127.0.0.1:46715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1002055 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1002055 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1002056 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1002057 T5552 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d17a73a name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1002057 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1002062 T5475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1003066 T5475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46747_
   [junit4]   2> 1003067 T5475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46747_
   [junit4]   2> 1003071 T5475 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1003075 T5475 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 1003076 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1004321 T5475 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1004322 T5475 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1004327 T5475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1004328 T5475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:46754
   [junit4]   2> 1004330 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1004331 T5475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1004331 T5475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-006
   [junit4]   2> 1004331 T5475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-006/'
   [junit4]   2> 1004361 T5475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-006/solr.xml
   [junit4]   2> 1004413 T5475 oasc.CoreContainer.<init> New CoreContainer 355588980
   [junit4]   2> 1004414 T5475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-006/]
   [junit4]   2> 1004414 T5475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 1004415 T5475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 1004415 T5475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1004415 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1004415 T5475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1004416 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1004416 T5475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1004416 T5475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1004416 T5475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1004418 T5475 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 1004420 T5475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1004420 T5475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1004420 T5475 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1004420 T5475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46715/solr
   [junit4]   2> 1004420 T5475 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1004421 T5475 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1004421 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1004423 T5565 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43b014f6 name:ZooKeeperConnection Watcher:127.0.0.1:46715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1004424 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1004424 T5475 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1004426 T5475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1004427 T5567 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@426b8dbd name:ZooKeeperConnection Watcher:127.0.0.1:46715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1004428 T5475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1004432 T5475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1005436 T5475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46754_
   [junit4]   2> 1005437 T5475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46754_
   [junit4]   2> 1005441 T5475 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1005446 T5475 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 1005446 T5475 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1005579 T5517 oasha.CollectionsHandler.handleCreateAction Creating Collection : wt=javabin&collection.configName=conf1&router.field=myOwnField&name=solrj_collection&replicationFactor=2&numShards=2&version=2&action=CREATE
   [junit4]   2> 1005580 T5508 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1005581 T5501 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1005582 T5571 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1005584 T5571 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1005584 T5571 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1005586 T5508 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1005589 T5500 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 1005589 T5500 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 1005589 T5500 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 1005591 T5508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005591 T5568 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005591 T5538 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005591 T5523 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005591 T5509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005591 T5553 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005687 T5571 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1005688 T5571 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:46737_
   [junit4]   2> 1005689 T5571 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:46747_
   [junit4]   2> 1005691 T5571 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:46723_
   [junit4]   2> 1005694 T5571 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:46754_
   [junit4]   2> 1005821 T5528 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&collection.configName=conf1&name=solrj_collection_shard1_replica1&shard=shard1&qt=/admin/cores&numShards=2&version=2&action=CREATE&collection=solrj_collection
   [junit4]   2> 1005822 T5489 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&collection.configName=conf1&name=solrj_collection_shard2_replica1&shard=shard2&qt=/admin/cores&numShards=2&version=2&action=CREATE&collection=solrj_collection
   [junit4]   2> 1005824 T5542 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&collection.configName=conf1&name=solrj_collection_shard1_replica2&shard=shard1&qt=/admin/cores&numShards=2&version=2&action=CREATE&collection=solrj_collection
   [junit4]   2> 1005825 T5557 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&collection.configName=conf1&name=solrj_collection_shard2_replica2&shard=shard2&qt=/admin/cores&numShards=2&version=2&action=CREATE&collection=solrj_collection
   [junit4]   2> 1005825 T5528 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 1005825 T5489 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 1005826 T5528 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1005826 T5489 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1005826 T5508 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1005827 T5542 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 1005828 T5542 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1005828 T5557 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 1005829 T5557 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1005830 T5500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:46737_",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46737",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"solrj_collection"}
   [junit4]   2> 1005832 T5500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:46723_",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46723",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"solrj_collection"}
   [junit4]   2> 1005833 T5508 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1005834 T5500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:46747_",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46747",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"solrj_collection"}
   [junit4]   2> 1005836 T5500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:46754_",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:46754",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"solrj_collection"}
   [junit4]   2> 1005838 T5508 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1005838 T5553 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005839 T5538 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005839 T5508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005839 T5509 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005838 T5523 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1005838 T5568 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1006827 T5528 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1006827 T5489 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1006828 T5528 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1006828 T5542 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1006829 T5528 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1006829 T5542 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1006828 T5489 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1006830 T5542 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1006830 T5489 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1006830 T5557 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1006830 T5542 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1006829 T5528 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1006831 T5557 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1006831 T5489 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1006832 T5557 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1006832 T5557 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1006832 T5528 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1006832 T5542 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1006833 T5557 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1006833 T5528 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-004/solrj_collection_shard1_replica1/'
   [junit4]   2> 1006833 T5489 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1006834 T5557 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-006/solrj_collection_shard2_replica2/'
   [junit4]   2> 1006833 T5542 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-005/solrj_collection_shard1_replica2/'
   [junit4]   2> 1006834 T5489 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001/tempDir-002/solrj_collection_shard2_replica1/'
   [junit4]   2> 1006920 T5528 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1006926 T5557 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1006931 T5542 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1006936 T5489 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1006975 T5528 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1006977 T5528 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1006980 T5557 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1006982 T5557 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1006984 T5542 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1006985 T5542 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1006988 T5528 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1006990 T5489 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1006991 T5489 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1006993 T5557 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1006996 T5542 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1007027 T5489 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1007481 T5528 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1007481 T5542 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1007484 T5557 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1007487 T5489 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1007491 T5528 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1007492 T5557 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1007493 T5542 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1007493 T5489 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1007494 T5528 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1007495 T5557 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1007496 T5542 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1007497 T5489 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1007506 T5528 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1007507 T5542 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1007507 T5557 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1007507 T5489 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1007511 T5528 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1007511 T5542 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1007512 T5489 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 100

[...truncated too long message...]

.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5912, name=Thread-3110, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection4_3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([29DDB92365BE42DE]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5911, name=Thread-3109, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection3_3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([29DDB92365BE42DE]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5909, name=Thread-3107, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection1_3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([29DDB92365BE42DE]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5908, name=Thread-3106, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection0_3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([29DDB92365BE42DE]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5910, name=Thread-3108, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection2_3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([29DDB92365BE42DE]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]   2> 1882536 T5475 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-29DDB92365BE42DE-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {_version_=OrdsLucene41(blocksize=128), id=FST41, a_t=OrdsLucene41(blocksize=128), multiDefault=PostingsFormat(name=Lucene41VarGapFixedInterval), a_i1=OrdsLucene41(blocksize=128), other_tl1=OrdsLucene41(blocksize=128), timestamp=OrdsLucene41(blocksize=128), text=PostingsFormat(name=SimpleText), range_facet_l=OrdsLucene41(blocksize=128), intDefault=OrdsLucene41(blocksize=128)}, docValues:{timestamp=DocValuesFormat(name=Lucene410)}, sim=DefaultSimilarity, locale=et_EE, timezone=Canada/Yukon
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=197527256,total=471859200
   [junit4]   2> NOTE: All tests run in this JVM: [ClusterStateUpdateTest, TestSchemaVersionResource, DistributedTermsComponentTest, TestManagedSchemaDynamicFieldResource, TestSort, TestReplicationHandlerBackup, TestCursorMarkWithoutUniqueKey, TestPivotHelperCode, TestJoin, DocValuesMultiTest, TestSolrXmlPersistor, FieldMutatingUpdateProcessorTest, TestDefaultSearchFieldResource, BlockDirectoryTest, MigrateRouteKeyTest, TestSimpleQParserPlugin, UUIDFieldTest, TestCSVLoader, TermVectorComponentDistributedTest, DistributedSuggestComponentTest, TestOrdValues, TestSearchPerf, DocValuesMissingTest, SolrInfoMBeanTest, TestCloudSchemaless, DOMUtilTest, DocumentAnalysisRequestHandlerTest, TestBadConfig, SolrIndexSplitterTest, TestDistributedGrouping, DocExpirationUpdateProcessorFactoryTest, SOLR749Test, AnalysisErrorHandlingTest, SuggesterTest, ShardRoutingTest, TestIndexSearcher, SolrPluginUtilsTest, ChangedSchemaMergeTest, TestPhraseSuggestions, SuggesterTSTTest, TestCoreDiscovery, DistributedExpandComponentTest, UniqFieldsUpdateProcessorFactoryTest, TestSolrXml, TimeZoneUtilsTest, MultiThreadedOCPTest, ZkCLITest, ReplicationFactorTest, TestLMJelinekMercerSimilarityFactory, TestCSVResponseWriter, TestDocSet, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J3 in 887.74s, 1 test, 1 error <<< FAILURES!

[...truncated 750 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 435 suites, 1782 tests, 1 error, 1 failure, 54 ignored (10 assumptions)

Total time: 172 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 596304039 bytes
Compression is 0.0%
Took 1 min 39 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 644 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/644/

1 tests failed.
REGRESSION:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11780 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/init-core-data-001
   [junit4]   2> 296065 T1640 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 296065 T1640 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 296071 T1640 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 296072 T1640 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 296073 T1641 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 296172 T1640 oasc.ZkTestServer.run start zk server on port:17700
   [junit4]   2> 296173 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 296174 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296177 T1647 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@424402c7 name:ZooKeeperConnection Watcher:127.0.0.1:17700 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296178 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296178 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 296178 T1640 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 296181 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 296182 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296183 T1649 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78a92bfb name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296183 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296184 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 296184 T1640 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 296186 T1640 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 296188 T1640 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 296190 T1640 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 296193 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 296193 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 296197 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 296198 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 296203 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 296204 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 296206 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 296207 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 296209 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 296210 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 296212 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 296213 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 296215 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 296216 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 296218 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 296219 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 296221 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 296222 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 296224 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 296225 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 296227 T1640 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 296228 T1640 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 296231 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 296232 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 296233 T1651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25d607ab name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 296233 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 296234 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using old style solr.xml
   [junit4]   2> 297446 T1640 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 297452 T1640 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 297455 T1640 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:17704
   [junit4]   2> 297458 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 297458 T1640 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 297458 T1640 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-002
   [junit4]   2> 297459 T1640 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-002/'
   [junit4]   2> 297488 T1640 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-002/solr.xml
   [junit4]   2> 297542 T1640 oasc.CoreContainer.<init> New CoreContainer 1943541385
   [junit4]   2> 297543 T1640 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-002/]
   [junit4]   2> 297544 T1640 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 297544 T1640 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 297544 T1640 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 297544 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 297544 T1640 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 297545 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 297545 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 297545 T1640 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 297545 T1640 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 297548 T1640 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 297550 T1640 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 297550 T1640 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 297550 T1640 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 297550 T1640 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:17700/solr
   [junit4]   2> 297551 T1640 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 297551 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 297552 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297553 T1662 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3dc82b58 name:ZooKeeperConnection Watcher:127.0.0.1:17700 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297554 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297554 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 297556 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297556 T1664 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49d85df9 name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297557 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297558 T1640 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 297559 T1640 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 297561 T1640 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 297563 T1640 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 297565 T1640 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 297567 T1640 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 297568 T1640 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:17704_
   [junit4]   2> 297568 T1640 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:17704_
   [junit4]   2> 297570 T1640 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 297571 T1640 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 297572 T1640 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 297573 T1640 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:17704_
   [junit4]   2> 297574 T1640 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 297575 T1640 oasc.Overseer.start Overseer (id=92548045937115140-127.0.0.1:17704_-n_0000000000) starting
   [junit4]   2> 297577 T1640 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 297582 T1640 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 297583 T1666 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 297583 T1640 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 297585 T1640 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 297586 T1640 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297588 T1665 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 297591 T1640 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 297591 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 297595 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 297596 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297597 T1670 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c4e6697 name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297597 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297598 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 297598 T1640 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 297601 T1640 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 297601 T1640 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 297602 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 297602 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 297603 T1672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1828c113 name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 297603 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 297603 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 297605 T1673 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 297608 T1665 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 297608 T1665 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 297608 T1665 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 297610 T1665 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 297611 T1673 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 297612 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 298572 T1640 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 298572 T1640 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 298578 T1640 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 298581 T1640 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:17709
   [junit4]   2> 298583 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 298584 T1640 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 298584 T1640 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003
   [junit4]   2> 298584 T1640 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/'
   [junit4]   2> 298614 T1640 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/solr.xml
   [junit4]   2> 298668 T1640 oasc.CoreContainer.<init> New CoreContainer 53740882
   [junit4]   2> 298669 T1640 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/]
   [junit4]   2> 298670 T1640 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 298670 T1640 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 298670 T1640 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 298670 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 298670 T1640 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 298671 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 298671 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 298671 T1640 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 298671 T1640 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 298674 T1640 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 298676 T1640 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 298676 T1640 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 298676 T1640 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 298677 T1640 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:17700/solr
   [junit4]   2> 298677 T1640 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 298677 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 298678 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 298679 T1685 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e9f32f9 name:ZooKeeperConnection Watcher:127.0.0.1:17700 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 298680 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 298680 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 298682 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 298683 T1687 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b2e31f3 name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 298683 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 298687 T1640 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 299692 T1640 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:17709_
   [junit4]   2> 299693 T1640 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:17709_
   [junit4]   2> 299697 T1640 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 299702 T1640 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 299702 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 300860 T1640 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 300862 T1640 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 300867 T1640 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 300869 T1640 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:17712
   [junit4]   2> 300871 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 300871 T1640 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 300872 T1640 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004
   [junit4]   2> 300872 T1640 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/'
   [junit4]   2> 300903 T1640 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/solr.xml
   [junit4]   2> 300977 T1640 oasc.CoreContainer.<init> New CoreContainer 1583387975
   [junit4]   2> 300977 T1640 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/]
   [junit4]   2> 300978 T1640 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 300978 T1640 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 300978 T1640 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 300978 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 300979 T1640 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 300979 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 300979 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 300979 T1640 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 300980 T1640 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 300982 T1640 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 300984 T1640 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 300984 T1640 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 300984 T1640 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 300985 T1640 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:17700/solr
   [junit4]   2> 300985 T1640 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 300985 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 300986 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 300987 T1700 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18a086a3 name:ZooKeeperConnection Watcher:127.0.0.1:17700 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 300988 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 300988 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 300990 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 300991 T1702 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a2ecdf6 name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 300991 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 300996 T1640 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 302002 T1640 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:17712_
   [junit4]   2> 302003 T1640 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:17712_
   [junit4]   2> 302008 T1640 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 302015 T1640 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 302016 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 303625 T1640 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 303626 T1640 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 303635 T1640 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 303639 T1640 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:17715
   [junit4]   2> 303643 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 303643 T1640 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 303644 T1640 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005
   [junit4]   2> 303644 T1640 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/'
   [junit4]   2> 303687 T1640 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/solr.xml
   [junit4]   2> 303763 T1640 oasc.CoreContainer.<init> New CoreContainer 751794087
   [junit4]   2> 303764 T1640 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/]
   [junit4]   2> 303765 T1640 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 303765 T1640 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 303765 T1640 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 303766 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 303766 T1640 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 303766 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 303767 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 303767 T1640 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 303767 T1640 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 303771 T1640 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 303773 T1640 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 303774 T1640 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 303774 T1640 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 303774 T1640 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:17700/solr
   [junit4]   2> 303775 T1640 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 303775 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 303776 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 303778 T1715 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ab9c308 name:ZooKeeperConnection Watcher:127.0.0.1:17700 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 303778 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 303779 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 303781 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 303782 T1717 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22841632 name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 303782 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 303788 T1640 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 304794 T1640 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:17715_
   [junit4]   2> 304795 T1640 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:17715_
   [junit4]   2> 304799 T1640 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 304805 T1640 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 304806 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 306296 T1640 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 306297 T1640 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 306304 T1640 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 306307 T1640 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:17718
   [junit4]   2> 306311 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 306311 T1640 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 306312 T1640 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006
   [junit4]   2> 306312 T1640 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/'
   [junit4]   2> 306358 T1640 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/solr.xml
   [junit4]   2> 306439 T1640 oasc.CoreContainer.<init> New CoreContainer 1556831446
   [junit4]   2> 306440 T1640 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/]
   [junit4]   2> 306441 T1640 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 306442 T1640 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 306442 T1640 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 306442 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 306443 T1640 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 306443 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 306443 T1640 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 306444 T1640 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 306444 T1640 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 306448 T1640 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=3400000&connTimeout=45000&retry=false
   [junit4]   2> 306451 T1640 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 306451 T1640 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 306451 T1640 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 306452 T1640 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:17700/solr
   [junit4]   2> 306452 T1640 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 306452 T1640 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 306453 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 306455 T1730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4d0832e0 name:ZooKeeperConnection Watcher:127.0.0.1:17700 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 306456 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 306456 T1640 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 306458 T1640 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 306459 T1732 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69704cd0 name:ZooKeeperConnection Watcher:127.0.0.1:17700/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 306459 T1640 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 306466 T1640 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 307472 T1640 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:17718_
   [junit4]   2> 307473 T1640 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:17718_
   [junit4]   2> 307477 T1640 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 307481 T1640 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 307481 T1640 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 307628 T1678 oasha.CollectionsHandler.handleCreateAction Creating Collection : router.field=myOwnField&action=CREATE&numShards=2&replicationFactor=2&version=2&name=solrj_collection&wt=javabin&collection.configName=conf1
   [junit4]   2> 307629 T1673 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307630 T1666 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 307635 T1736 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 307640 T1736 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 307640 T1736 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 307644 T1673 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307647 T1665 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 307647 T1665 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 307647 T1665 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 307650 T1673 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307650 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307650 T1718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307650 T1733 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307650 T1703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307650 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307744 T1736 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 307745 T1736 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:17718_
   [junit4]   2> 307746 T1736 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:17712_
   [junit4]   2> 307748 T1736 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:17715_
   [junit4]   2> 307750 T1736 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:17709_
   [junit4]   2> 307874 T1722 oasha.CoreAdminHandler.handleCreateAction core create command action=CREATE&numShards=2&collection=solrj_collection&qt=/admin/cores&version=2&name=solrj_collection_shard1_replica1&shard=shard1&wt=javabin&collection.configName=conf1
   [junit4]   2> 307877 T1722 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 307878 T1673 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307878 T1722 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 307883 T1665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "node_name":"127.0.0.1:17718_",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:17718",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1"}
   [junit4]   2> 307883 T1707 oasha.CoreAdminHandler.handleCreateAction core create command action=CREATE&numShards=2&collection=solrj_collection&qt=/admin/cores&version=2&name=solrj_collection_shard2_replica1&shard=shard2&wt=javabin&collection.configName=conf1
   [junit4]   2> 307883 T1680 oasha.CoreAdminHandler.handleCreateAction core create command action=CREATE&numShards=2&collection=solrj_collection&qt=/admin/cores&version=2&name=solrj_collection_shard2_replica2&shard=shard2&wt=javabin&collection.configName=conf1
   [junit4]   2> 307886 T1707 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 307887 T1680 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 307887 T1693 oasha.CoreAdminHandler.handleCreateAction core create command action=CREATE&numShards=2&collection=solrj_collection&qt=/admin/cores&version=2&name=solrj_collection_shard1_replica2&shard=shard1&wt=javabin&collection.configName=conf1
   [junit4]   2> 307887 T1707 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 307887 T1680 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 307887 T1673 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307889 T1665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "node_name":"127.0.0.1:17715_",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:17715",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2"}
   [junit4]   2> 307889 T1693 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 307890 T1693 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 307892 T1665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "node_name":"127.0.0.1:17709_",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:17709",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2"}
   [junit4]   2> 307894 T1673 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 307895 T1665 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "node_name":"127.0.0.1:17712_",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:17712",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1"}
   [junit4]   2> 307998 T1673 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307998 T1674 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307998 T1703 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307998 T1733 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307998 T1688 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 307998 T1718 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 308879 T1722 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 308880 T1722 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 308881 T1722 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 308881 T1722 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 308882 T1722 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 308882 T1722 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/solrj_collection_shard1_replica1/'
   [junit4]   2> 308889 T1680 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 308889 T1707 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 308889 T1680 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 308890 T1707 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 308890 T1680 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 308890 T1693 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 308890 T1707 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 308891 T1693 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 308891 T1680 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 308892 T1693 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 308891 T1707 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 308892 T1693 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 308893 T1680 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 308894 T1680 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/solrj_collection_shard2_replica2/'
   [junit4]   2> 308894 T1693 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 308894 T1707 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 308894 T1693 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/solrj_collection_shard1_replica2/'
   [junit4]   2> 308895 T1707 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/solrj_collection_shard2_replica1/'
   [junit4]   2> 308967 T1722 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 308980 T1680 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 308985 T1707 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 308991 T1693 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 309028 T1722 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 309030 T1722 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 309041 T1680 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 309042 T1680 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 309043 T1722 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 309045 T1707 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 309047 T1707 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 309051 T1693 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 309052 T1693 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 309056 T1680 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 309060 T1707 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 309065 T1693 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 309700 T1680 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 309705 T1722 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 309712 T1693 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 309716 T1680 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 309718 T1722 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 309721 T1680 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 309723 T1693 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 309724 T1722 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 309727 T1693 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 309730 T1707 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 309740 T1680 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309741 T1707 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 309744 T1722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309745 T1707 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 309746 T1693 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309747 T1680 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309751 T1722 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309752 T1693 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309753 T1680 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309754 T1680 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309756 T1680 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309757 T1680 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309758 T1680 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309759 T1680 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309759 T1680 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 309760 T1680 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 309760 T1722 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309760 T1680 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/solrj_collection_shard2_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/solrj_collection_shard2_replica2/data/
   [junit4]   2> 309760 T1693 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309761 T1722 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309761 T1680 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@232df1e9
   [junit4]   2> 309763 T1693 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309762 T1722 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309763 T1693 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309764 T1680 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/solrj_collection_shard2_replica2/data
   [junit4]   2> 309764 T1722 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309764 T1680 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 309764 T1693 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309766 T1722 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309766 T1707 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309766 T1680 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 309767 T1693 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309766 T1722 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309768 T1680 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-003/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 309768 T1693 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309769 T1680 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8364886009501442]
   [junit4]   2> 309768 T1722 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 309769 T1693 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 309770 T1722 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 309770 T1680 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-4ab5a1c1); maxCacheMB=0.8466796875 maxMergeSizeMB=0.265625)),segFN=segments_1,generation=1}
   [junit4]   2> 309770 T1693 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 309771 T1680 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 309771 T1722 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/solrj_collection_shard1_replica1/data/
   [junit4]   2> 309784 T1693 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/solrj_collection_shard1_replica2/data/
   [junit4]   2> 309785 T1722 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@232df1e9
   [junit4]   2> 309786 T1707 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 309785 T1693 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@232df1e9
   [junit4]   2> 309787 T1722 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/solrj_collection_shard1_replica1/data
   [junit4]   2> 309788 T1693 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/solrj_collection_shard1_replica2/data
   [junit4]   2> 309788 T1722 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 309788 T1693 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 309789 T1722 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 309789 T1693 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 309790 T1722 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-006/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 309790 T1693 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-004/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 309791 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 309791 T1693 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8364886009501442]
   [junit4]   2> 309792 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 309791 T1722 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8364886009501442]
   [junit4]   2> 309793 T1693 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-7ab72e51); maxCacheMB=0.8466796875 maxMergeSizeMB=0.265625)),segFN=segments_1,generation=1}
   [junit4]   2> 309792 T1680 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 309794 T1722 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-388859e4); maxCacheMB=0.8466796875 maxMergeSizeMB=0.265625)),segFN=segments_1,generation=1}
   [junit4]   2> 309792 T1707 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309794 T1722 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 309794 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309795 T1707 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309793 T1693 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 309796 T1707 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309796 T1680 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309797 T1707 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 309797 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 309798 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309798 T1707 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 309799 T1680 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309799 T1707 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 309799 T1680 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 309800 T1707 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 309801 T1707 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 309801 T1680 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309801 T1707 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/solrj_collection_shard2_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/solrj_collection_shard2_replica1/data/
   [junit4]   2> 309802 T1680 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309802 T1707 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@232df1e9
   [junit4]   2> 309802 T1680 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309803 T1722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 309803 T1693 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 309803 T1722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 309803 T1707 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/solrj_collection_shard2_replica1/data
   [junit4]   2> 309803 T1680 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 309804 T1707 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 309805 T1680 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 309804 T1722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 309804 T1693 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 309806 T1722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309805 T1680 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 309805 T1707 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 309807 T1680 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 309807 T1722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309808 T1680 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 309806 T1693 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 309809 T1680 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 309808 T1722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 309808 T1707 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001/tempDir-005/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 309810 T1680 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 309810 T1707 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=22, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8364886009501442]
   [junit4]   2> 309809 T1693 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309811 T1680 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 309811 T1707 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-6e9ee014); maxCacheMB=0.8466796875 maxMergeSizeMB=0.265625)),segFN=segments_1,generation=1}
   [junit4]   2> 309810 T1722 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309812 T1707 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 309812 T1722 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309813 T1722 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 309811 T1693 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309814 T1693 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 309814 T1722 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309814 T1693 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309815 T1722 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309815 T1693 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309815 T1722 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309816 T1693 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 309816 T1722 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 309817 T1693 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309817 T1722 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 309818 T1722 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 309818 T1693 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309818 T1722 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 309819 T1693 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309819 T1722 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 309819 T1693 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 309819 T1707 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 309820 T1722 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 309820 T1707 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 309820 T1693 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 309821 T1707 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 309821 T1722 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 309821 T1707 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309821 T1693 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 309822 T1722 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 309822 T1707 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 309823 T1693 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 309823 T1707 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 309824 T1693 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 309825 T1707 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309825 T1693 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 309825 T1707 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 309826 T1707 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 309826 T1693 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 309827 T1707 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309827 T1693 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 309828 T1707 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309829 T1707 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 309830 T1707 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 309830 T1707 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 309830 T1707 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 309831 T1707 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 309832 T1707 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 309832 T1707 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 309833 T1707 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 309834 T1707 oasc.RequestHandlers.initHandlersFr

[...truncated too long message...]

    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1632)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection0_1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 6 moreThrowable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3083, name=Thread-1549, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:17712, https://127.0.0.1:17718, https://127.0.0.1:17715, https://127.0.0.1:17704, https://127.0.0.1:17709]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F9CEC87936445634]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1056)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:17712, https://127.0.0.1:17718, https://127.0.0.1:17715, https://127.0.0.1:17704, https://127.0.0.1:17709]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1632)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection1_1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 6 moreThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=3085, name=Thread-1551, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:17712, https://127.0.0.1:17718, https://127.0.0.1:17715, https://127.0.0.1:17704, https://127.0.0.1:17709]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([F9CEC87936445634]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1056)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:17712, https://127.0.0.1:17718, https://127.0.0.1:17715, https://127.0.0.1:17704, https://127.0.0.1:17709]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1632)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection3_1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 6 more
   [junit4]   2> 1214412 T1640 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-F9CEC87936445634-001
   [junit4]   2> 918354 T1639 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=456), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=456)), sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=ca_ES, timezone=Asia/Nicosia
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=100090592,total=413663232
   [junit4]   2> NOTE: All tests run in this JVM: [StatelessScriptUpdateProcessorFactoryTest, TestBadConfig, ClusterStateUpdateTest, ResponseLogComponentTest, SOLR749Test, StandardRequestHandlerTest, TestInfoStreamLogging, UnloadDistributedZkTest, CoreContainerCoreInitFailuresTest, BadIndexSchemaTest, RequestHandlersTest, TestDistributedGrouping, TestDynamicFieldResource, InfoHandlerTest, TestSolrQueryParserDefaultOperatorResource, TestCopyFieldCollectionResource, TestLRUCache, TestReRankQParserPlugin, ZkCLITest, BadCopyFieldTest, SpellCheckComponentTest, OverriddenZkACLAndCredentialsProvidersTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J3 in 919.15s, 1 test, 1 error <<< FAILURES!

[...truncated 975 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 1 error, 54 ignored (10 assumptions)

Total time: 207 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 695787949 bytes
Compression is 0.0%
Took 1 min 57 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 643 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/643/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch

Error Message:
no exception matching expected: 400: Request took too long during query expansion. Terminating request.

Stack Trace:
java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request.
	at __randomizedtesting.SeedInfo.seed([6E87D73A61D675B3:EF6159221689158F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


REGRESSION:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1412090974829,generation=2,filelist=[_20.fdt, _20.fdx, _20.fnm, _20.nvd, _20.nvm, _20.si, _20_Asserting_0.doc, _20_Asserting_0.tim, _20_Asserting_0.tip, _41.cfe, _41.cfs, _41.si, _62.cfe, _62.cfs, _62.si, _83.cfe, _83.cfs, _83.si, _a4.cfe, _a4.cfs, _a4.si, _bn.cfe, _bn.cfs, _bn.si, _bo.cfe, _bo.cfs, _bo.si, _bp.cfe, _bp.cfs, _bp.si, _bq.cfe, _bq.cfs, _bq.si, _br.cfe, _br.cfs, _br.si, _bs.cfe, _bs.cfs, _bs.si, _bt.cfe, _bt.cfs, _bt.si, _bu.cfe, _bu.cfs, _bu.si, _bv.cfe, _bv.cfs, _bv.si, segments_2]}]> but was:<[{indexVersion=1412090974829,generation=3,filelist=[_20.fdt, _20.fdx, _20.fnm, _20.nvd, _20.nvm, _20.si, _20_Asserting_0.doc, _20_Asserting_0.tim, _20_Asserting_0.tip, _41.cfe, _41.cfs, _41.si, _62.cfe, _62.cfs, _62.si, _83.cfe, _83.cfs, _83.si, _a4.cfe, _a4.cfs, _a4.si, _bw.cfe, _bw.cfs, _bw.si, segments_3]}, {indexVersion=1412090974829,generation=2,filelist=[_20.fdt, _20.fdx, _20.fnm, _20.nvd, _20.nvm, _20.si, _20_Asserting_0.doc, _20_Asserting_0.tim, _20_Asserting_0.tip, _41.cfe, _41.cfs, _41.si, _62.cfe, _62.cfs, _62.si, _83.cfe, _83.cfs, _83.si, _a4.cfe, _a4.cfs, _a4.si, _bn.cfe, _bn.cfs, _bn.si, _bo.cfe, _bo.cfs, _bo.si, _bp.cfe, _bp.cfs, _bp.si, _bq.cfe, _bq.cfs, _bq.si, _br.cfe, _br.cfs, _br.si, _bs.cfe, _bs.cfs, _bs.si, _bt.cfe, _bt.cfs, _bt.si, _bu.cfe, _bu.cfs, _bu.si, _bv.cfe, _bv.cfs, _bv.si, segments_2]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1412090974829,generation=2,filelist=[_20.fdt, _20.fdx, _20.fnm, _20.nvd, _20.nvm, _20.si, _20_Asserting_0.doc, _20_Asserting_0.tim, _20_Asserting_0.tip, _41.cfe, _41.cfs, _41.si, _62.cfe, _62.cfs, _62.si, _83.cfe, _83.cfs, _83.si, _a4.cfe, _a4.cfs, _a4.si, _bn.cfe, _bn.cfs, _bn.si, _bo.cfe, _bo.cfs, _bo.si, _bp.cfe, _bp.cfs, _bp.si, _bq.cfe, _bq.cfs, _bq.si, _br.cfe, _br.cfs, _br.si, _bs.cfe, _bs.cfs, _bs.si, _bt.cfe, _bt.cfs, _bt.si, _bu.cfe, _bu.cfs, _bu.si, _bv.cfe, _bv.cfs, _bv.si, segments_2]}]> but was:<[{indexVersion=1412090974829,generation=3,filelist=[_20.fdt, _20.fdx, _20.fnm, _20.nvd, _20.nvm, _20.si, _20_Asserting_0.doc, _20_Asserting_0.tim, _20_Asserting_0.tip, _41.cfe, _41.cfs, _41.si, _62.cfe, _62.cfs, _62.si, _83.cfe, _83.cfs, _83.si, _a4.cfe, _a4.cfs, _a4.si, _bw.cfe, _bw.cfs, _bw.si, segments_3]}, {indexVersion=1412090974829,generation=2,filelist=[_20.fdt, _20.fdx, _20.fnm, _20.nvd, _20.nvm, _20.si, _20_Asserting_0.doc, _20_Asserting_0.tim, _20_Asserting_0.tip, _41.cfe, _41.cfs, _41.si, _62.cfe, _62.cfs, _62.si, _83.cfe, _83.cfs, _83.si, _a4.cfe, _a4.cfs, _a4.si, _bn.cfe, _bn.cfs, _bn.si, _bo.cfe, _bo.cfs, _bo.si, _bp.cfe, _bp.cfs, _bp.si, _bq.cfe, _bq.cfs, _bq.si, _br.cfe, _br.cfs, _br.si, _bs.cfe, _bs.cfs, _bs.si, _bt.cfe, _bt.cfs, _bt.si, _bu.cfe, _bu.cfs, _bu.si, _bv.cfe, _bv.cfs, _bv.si, segments_2]}]>
	at __randomizedtesting.SeedInfo.seed([6E87D73A61D675B3:4B50CC0A119E7BB0]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1175)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12440 lines...]
   [junit4] Suite: org.apache.solr.cloud.CloudExitableDirectoryReaderTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/init-core-data-001
   [junit4]   2> 2509181 T771928 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2509181 T771928 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2509185 T771928 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2509186 T771928 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2509187 T771929 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2509286 T771928 oasc.ZkTestServer.run start zk server on port:23386
   [junit4]   2> 2509287 T771928 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2509288 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2509293 T771935 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fa9ddfd name:ZooKeeperConnection Watcher:127.0.0.1:23386 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2509293 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2509293 T771928 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2509294 T771928 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2509297 T771928 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2509298 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2509299 T771937 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@522c9a17 name:ZooKeeperConnection Watcher:127.0.0.1:23386/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2509300 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2509300 T771928 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2509300 T771928 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2509303 T771928 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2509305 T771928 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2509306 T771928 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2509308 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2509309 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2509312 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2509312 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2509416 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2509417 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2509419 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2509420 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2509422 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2509423 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2509429 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2509430 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2509432 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2509433 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2509435 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2509436 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2509438 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2509438 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2509440 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2509441 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2509443 T771928 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2509443 T771928 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2511214 T771928 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2511218 T771928 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:23391
   [junit4]   2> 2511218 T771928 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2511219 T771928 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2511219 T771928 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002
   [junit4]   2> 2511219 T771928 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002/'
   [junit4]   2> 2511264 T771928 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002/solr.xml
   [junit4]   2> 2511323 T771928 oasc.CoreContainer.<init> New CoreContainer 2137176113
   [junit4]   2> 2511324 T771928 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002/]
   [junit4]   2> 2511324 T771928 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2511325 T771928 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2511325 T771928 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2511325 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2511325 T771928 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2511326 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2511326 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2511326 T771928 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2511326 T771928 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2511327 T771928 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2511327 T771928 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2511327 T771928 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2511327 T771928 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2511328 T771928 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:23386/solr
   [junit4]   2> 2511328 T771928 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2511328 T771928 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2511329 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2511330 T771948 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a2ada4d name:ZooKeeperConnection Watcher:127.0.0.1:23386 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2511331 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2511331 T771928 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2511333 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2511334 T771950 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38978d84 name:ZooKeeperConnection Watcher:127.0.0.1:23386/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2511334 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2511335 T771928 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2511337 T771928 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2511339 T771928 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2511341 T771928 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2511343 T771928 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2511345 T771928 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2511346 T771928 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23391_
   [junit4]   2> 2511347 T771928 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23391_
   [junit4]   2> 2511348 T771928 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2511350 T771928 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2511351 T771928 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2511352 T771928 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:23391_
   [junit4]   2> 2511353 T771928 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2511354 T771928 oasc.Overseer.start Overseer (id=92542794662215683-127.0.0.1:23391_-n_0000000000) starting
   [junit4]   2> 2511356 T771928 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2511362 T771928 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2511362 T771952 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2511363 T771928 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2511364 T771928 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2511365 T771928 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2511368 T771951 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2511371 T771954 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2511372 T771954 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2511372 T771954 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2511372 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2511374 T771951 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:23391_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:23391",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 2511375 T771951 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2511375 T771951 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 2511375 T771951 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2511377 T771955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2512373 T771954 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2512374 T771954 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2512375 T771954 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2512375 T771954 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2512375 T771954 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2512376 T771954 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002/collection1/'
   [junit4]   2> 2512378 T771954 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2512379 T771954 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2512379 T771954 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2512441 T771954 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2512487 T771954 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2512589 T771954 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2512599 T771954 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2512963 T771954 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2512975 T771954 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2512978 T771954 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2512988 T771954 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2512993 T771954 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2512996 T771954 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2512997 T771954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2512998 T771954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2512998 T771954 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2512999 T771954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2512999 T771954 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2512999 T771954 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2513000 T771954 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2513000 T771954 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-002/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/control/data/
   [junit4]   2> 2513000 T771954 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c4689ad
   [junit4]   2> 2513001 T771954 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/control/data
   [junit4]   2> 2513001 T771954 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/control/data/index/
   [junit4]   2> 2513001 T771954 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2513002 T771954 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/control/data/index
   [junit4]   2> 2513002 T771954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=6, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2513003 T771954 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-72ab63); maxCacheMB=0.9697265625 maxMergeSizeMB=0.228515625)),segFN=segments_1,generation=1}
   [junit4]   2> 2513003 T771954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2513007 T771954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2513008 T771954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2513008 T771954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2513008 T771954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2513009 T771954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2513009 T771954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2513009 T771954 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2513010 T771954 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2513010 T771954 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2513010 T771954 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2513011 T771954 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2513011 T771954 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2513012 T771954 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2513012 T771954 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2513012 T771954 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2513013 T771954 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2513013 T771954 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2513013 T771954 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2513014 T771954 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2513014 T771954 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2513033 T771954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2513035 T771954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2513036 T771954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2513038 T771954 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2513044 T771954 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2513044 T771954 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2513044 T771954 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=48, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6763237425722719]
   [junit4]   2> 2513045 T771954 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-72ab63); maxCacheMB=0.9697265625 maxMergeSizeMB=0.228515625)),segFN=segments_1,generation=1}
   [junit4]   2> 2513045 T771954 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2513046 T771954 oass.SolrIndexSearcher.<init> Opening Searcher@218bc114[collection1] main
   [junit4]   2> 2513046 T771954 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2513047 T771954 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2513047 T771954 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2513047 T771954 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2513048 T771954 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2513048 T771954 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2513048 T771954 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2513049 T771954 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2513049 T771954 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2513051 T771954 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2513051 T771954 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2513051 T771954 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2513054 T771956 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@218bc114[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2513057 T771954 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2513058 T771959 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:23391 collection:control_collection shard:shard1
   [junit4]   2> 2513058 T771928 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 2513058 T771928 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2513059 T771959 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2513059 T771928 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2513060 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2513061 T771961 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23c46a65 name:ZooKeeperConnection Watcher:127.0.0.1:23386/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2513061 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2513062 T771928 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2513063 T771928 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2513064 T771959 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2513065 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2513066 T771959 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2513066 T771959 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C37905 name=collection1 org.apache.solr.core.SolrCore@3930f267 url=http://127.0.0.1:23391/collection1 node=127.0.0.1:23391_ C37905_STATE=coll:control_collection core:collection1 props:{state=down, node_name=127.0.0.1:23391_, base_url=http://127.0.0.1:23391, core=collection1}
   [junit4]   2> 2513066 T771959 C37905 P23391 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:23391/collection1/
   [junit4]   2> 2513066 T771928 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2513066 T771959 C37905 P23391 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2513067 T771928 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 2513067 T771959 C37905 P23391 oasc.SyncStrategy.syncToMe http://127.0.0.1:23391/collection1/ has no replicas
   [junit4]   2> 2513067 T771928 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2513068 T771959 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:23391/collection1/ shard1
   [junit4]   2> 2513068 T771959 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2513068 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2513070 T771963 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ea6fd88 name:ZooKeeperConnection Watcher:127.0.0.1:23386/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2513070 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2513070 T771964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2513070 T771955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2513070 T771928 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2513073 T771965 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2513076 T771951 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 2513076 T771951 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2513076 T771951 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 2513078 T771951 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 2513079 T771955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2513079 T771964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2513084 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2513084 T771955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2513084 T771964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2513126 T771959 oasc.ZkController.register We are http://127.0.0.1:23391/collection1/ and leader is http://127.0.0.1:23391/collection1/
   [junit4]   2> 2513127 T771959 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:23391
   [junit4]   2> 2513127 T771959 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2513127 T771959 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2513127 T771959 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2513128 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2513131 T771951 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1:23391_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:23391",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 2513235 T771955 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2513235 T771964 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2514316 T771928 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2514317 T771928 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2514320 T771928 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:23403
   [junit4]   2> 2514320 T771928 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2514320 T771928 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2514321 T771928 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003
   [junit4]   2> 2514321 T771928 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003/'
   [junit4]   2> 2514354 T771928 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003/solr.xml
   [junit4]   2> 2514412 T771928 oasc.CoreContainer.<init> New CoreContainer 1751576970
   [junit4]   2> 2514413 T771928 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003/]
   [junit4]   2> 2514414 T771928 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2514414 T771928 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2514414 T771928 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2514414 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2514415 T771928 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2514415 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2514415 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2514415 T771928 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2514415 T771928 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2514416 T771928 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2514416 T771928 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2514416 T771928 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2514417 T771928 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2514417 T771928 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:23386/solr
   [junit4]   2> 2514417 T771928 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2514417 T771928 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2514418 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2514419 T771976 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e7bd092 name:ZooKeeperConnection Watcher:127.0.0.1:23386 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2514420 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2514420 T771928 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2514422 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2514423 T771978 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13dcf44d name:ZooKeeperConnection Watcher:127.0.0.1:23386/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2514423 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2514428 T771928 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2515433 T771928 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23403_
   [junit4]   2> 2515434 T771928 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23403_
   [junit4]   2> 2515437 T771928 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2515443 T771980 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2515444 T771980 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2515445 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2515445 T771980 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 2515446 T771980 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 2515446 T771980 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 2515447 T771980 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 2515447 T771951 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:23403_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:23403",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 2515447 T771980 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2515450 T771951 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 2515450 T771951 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2515454 T771951 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2515455 T771979 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2515455 T771979 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 2516449 T771980 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2516450 T771980 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2516450 T771980 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2516451 T771980 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2516451 T771980 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2516452 T771980 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003/collection1/'
   [junit4]   2> 2516454 T771980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2516454 T771980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2516455 T771980 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2516532 T771980 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2516578 T771980 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2516679 T771980 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2516689 T771980 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2517054 T771980 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2517066 T771980 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2517068 T771980 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2517079 T771980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2517084 T771980 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2517088 T771980 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2517089 T771980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2517090 T771980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2517090 T771980 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2517091 T771980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2517092 T771980 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2517092 T771980 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2517092 T771980 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2517092 T771980 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-003/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty1/
   [junit4]   2> 2517093 T771980 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c4689ad
   [junit4]   2> 2517093 T771980 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty1
   [junit4]   2> 2517094 T771980 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty1/index/
   [junit4]   2> 2517094 T771980 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2517094 T771980 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty1/index
   [junit4]   2> 2517095 T771980 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=6, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2517096 T771980 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-3ce79b4b); maxCacheMB=0.9697265625 maxMergeSizeMB=0.228515625)),segFN=segments_1,generation=1}
   [junit4]   2> 2517096 T771980 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2517101 T771980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2517101 T771980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2517102 T771980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2517102 T771980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2517102 T771980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2517102 T771980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2517103 T771980 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2517103 T771980 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2517103 T771980 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2517104 T771980 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2517104 T771980 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2517105 T771980 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2517106 T771980 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2517106 T771980 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2517106 T771980 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2517107 T771980 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2517107 T771980 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2517108 T771980 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2517108 T771980 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2517109 T771980 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2517127 T771980 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2517129 T771980 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2517131 T771980 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2517133 T771980 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2517137 T771980 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2517137 T771980 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2517138 T771980 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=48, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6763237425722719]
   [junit4]   2> 2517139 T771980 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-3ce79b4b); maxCacheMB=0.9697265625 maxMergeSizeMB=0.228515625)),segFN=segments_1,generation=1}
   [junit4]   2> 2517139 T771980 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2517139 T771980 oass.SolrIndexSearcher.<init> Opening Searcher@6d98311f[collection1] main
   [junit4]   2> 2517139 T771980 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2517140 T771980 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2517140 T771980 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2517141 T771980 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2517141 T771980 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2517141 T771980 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2517142 T771980 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2517142 T771980 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2517143 T771980 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2517143 T771980 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2517147 T771981 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d98311f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2517150 T771980 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2517151 T771984 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:23403 collection:collection1 shard:shard1
   [junit4]   2> 2517151 T771928 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2
   [junit4]   2> 2517152 T771928 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2517152 T771984 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2517156 T771984 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2517158 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2517158 T771984 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2517158 T771984 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C37906 name=collection1 org.apache.solr.core.SolrCore@498e273a url=http://127.0.0.1:23403/collection1 node=127.0.0.1:23403_ C37906_STATE=coll:collection1 core:collection1 props:{state=down, node_name=127.0.0.1:23403_, base_url=http://127.0.0.1:23403, core=collection1}
   [junit4]   2> 2517158 T771984 C37906 P23403 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:23403/collection1/
   [junit4]   2> 2517159 T771984 C37906 P23403 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2517159 T771984 C37906 P23403 oasc.SyncStrategy.syncToMe http://127.0.0.1:23403/collection1/ has no replicas
   [junit4]   2> 2517159 T771984 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:23403/collection1/ shard1
   [junit4]   2> 2517159 T771984 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2517162 T771951 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2517163 T771979 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2517163 T771979 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 2517168 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2517168 T771951 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2517169 T771979 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2517169 T771979 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 2517214 T771984 oasc.ZkController.register We are http://127.0.0.1:23403/collection1/ and leader is http://127.0.0.1:23403/collection1/
   [junit4]   2> 2517215 T771984 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:23403
   [junit4]   2> 2517215 T771984 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2517215 T771984 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2517215 T771984 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2517217 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2517219 T771984 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 2517219 T771951 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1:23403_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:23403",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 2517224 T771951 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2517224 T771979 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2517225 T771979 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 2518426 T771928 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2518427 T771928 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2518430 T771928 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:23412
   [junit4]   2> 2518430 T771928 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2518430 T771928 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2518431 T771928 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004
   [junit4]   2> 2518431 T771928 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004/'
   [junit4]   2> 2518461 T771928 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004/solr.xml
   [junit4]   2> 2518513 T771928 oasc.CoreContainer.<init> New CoreContainer 236749288
   [junit4]   2> 2518513 T771928 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004/]
   [junit4]   2> 2518514 T771928 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2518514 T771928 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2518514 T771928 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2518515 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2518515 T771928 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2518515 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2518515 T771928 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2518516 T771928 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2518516 T771928 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2518516 T771928 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2518516 T771928 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2518517 T771928 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2518517 T771928 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2518517 T771928 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:23386/solr
   [junit4]   2> 2518517 T771928 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2518518 T771928 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2518518 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2518520 T771995 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7194ad94 name:ZooKeeperConnection Watcher:127.0.0.1:23386 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2518520 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2518521 T771928 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2518523 T771928 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2518524 T771997 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@219de748 name:ZooKeeperConnection Watcher:127.0.0.1:23386/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2518524 T771928 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2518529 T771928 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2519533 T771928 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:23412_
   [junit4]   2> 2519534 T771928 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:23412_
   [junit4]   2> 2519537 T771928 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2519545 T771999 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2519545 T771999 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2519546 T771955 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2519547 T771999 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 2519547 T771999 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 2519548 T771999 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 2519549 T771951 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:23412_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:23412",
   [junit4]   2> 	  "core":"collection1"}
   [junit4]   2> 2519549 T771999 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 2519549 T771999 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2519551 T771951 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 2519552 T771951 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2519556 T771951 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 2519556 T771979 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2519556 T771998 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 2519557 T771979 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 2519557 T771998 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 2520550 T771999 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2520551 T771999 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2520552 T771999 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2520552 T771999 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2520553 T771999 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2520553 T771999 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004/collection1/'
   [junit4]   2> 2520554 T771999 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2520555 T771999 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2520556 T771999 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2520618 T771999 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2520665 T771999 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2520767 T771999 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2520777 T771999 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2521143 T771999 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2521155 T771999 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2521158 T771999 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2521168 T771999 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2521172 T771999 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2521176 T771999 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2521177 T771999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2521177 T771999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2521178 T771999 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2521178 T771999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2521179 T771999 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2521179 T771999 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2521179 T771999 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2521179 T771999 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-004/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty2/
   [junit4]   2> 2521180 T771999 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2c4689ad
   [junit4]   2> 2521180 T771999 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty2
   [junit4]   2> 2521181 T771999 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty2/index/
   [junit4]   2> 2521181 T771999 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2521181 T771999 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J2/temp/solr.cloud.CloudExitableDirectoryReaderTest-6E87D73A61D675B3-001/tempDir-001/jetty2/index
   [junit4]   2> 2521182 T771999 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=6, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2521182 T771999 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory(id=lucene-75a54dd6); maxCacheMB=0.9697265625 maxMergeSizeMB=0.228515625)),segFN=segments_1,generation=1}
   [junit4]   2> 2521183 T771999 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2521188 T771999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2521188 T771999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2521188 T771999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2521189 T771999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2521189 T771999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2521189 T771999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2521190 T771999 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2521190 T771999 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2521190 T771999 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2521191 T771999 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2521191 T771999 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2521192 T771999 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2521192 T771999 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2521193 T771999 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2521193 T771999 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2521194 T771999 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2521194 T771999 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2521194 T771999 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2521195 T771999 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2521196 T771999 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2521214 T771999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2521216 T771999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2521218 T771999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2521220 T771999 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2521224 T771999 oasu.CommitTracker

[...truncated too long message...]

nit4]   2> 2766539 T5971 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2766539 T5971 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2766540 T5971 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-001/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-001/collection1/data;done=false>>]
   [junit4]   2> 2766540 T5971 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-001/collection1/data
   [junit4]   2> 2766540 T5971 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-001/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-001/collection1/data/index;done=false>>]
   [junit4]   2> 2766541 T5971 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-001/collection1/data/index
   [junit4]   2> 2766541 T5971 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 2766923 T6897 oash.SnapPuller$1.run Poll disabled
   [junit4]   2> 2767924 T6897 oash.SnapPuller$1.run Poll disabled
   [junit4]   2> 2768924 T6897 oash.SnapPuller$1.run Poll disabled
   [junit4]   2> 2769924 T6897 oash.SnapPuller$1.run Poll disabled
   [junit4]   2> 2770924 T6897 oash.SnapPuller$1.run Poll disabled
   [junit4]   2> 2771538 T5971 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1918745402
   [junit4]   2> 2771539 T5971 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1a02e08e
   [junit4]   2> 2771539 T5971 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=5,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 2771540 T5971 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 2771540 T5971 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2771540 T5971 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2771542 T5971 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2771542 T5971 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2771542 T5971 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-002/collection1/data [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-002/collection1/data;done=false>>]
   [junit4]   2> 2771543 T5971 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-002/collection1/data
   [junit4]   2> 2771543 T5971 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-002/collection1/data/index [CachedDir<<refCount=0;path=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-002/collection1/data/index;done=false>>]
   [junit4]   2> 2771543 T5971 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001/solr-instance-002/collection1/data/index
   [junit4]   2> 2771544 T5971 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 2776551 T5971 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler-6E87D73A61D675B3-001
   [junit4]   2> 1180468 T5970 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 2781271 T6900 oash.SnapPuller.fetchLatestIndex ERROR Master at: http://127.0.0.1:61477/solr is not available. Index fetch failed. Exception: IOException occured when talking to server at: http://127.0.0.1:61477/solr
   [junit4]   2> 2781271 T6900 oasc.SolrException.log ERROR SnapPull failed :org.apache.lucene.store.AlreadyClosedException: Already closed
   [junit4]   2> 		at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:340)
   [junit4]   2> 		at org.apache.solr.handler.ReplicationHandler.loadReplicationProperties(ReplicationHandler.java:822)
   [junit4]   2> 		at org.apache.solr.handler.SnapPuller.logReplicationTimeAndConfFiles(SnapPuller.java:570)
   [junit4]   2> 		at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:512)
   [junit4]   2> 		at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
   [junit4]   2> 		at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:228)
   [junit4]   2> 	
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=sl_SI, timezone=America/Guadeloupe
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=151523744,total=362807296
   [junit4]   2> NOTE: All tests run in this JVM: [TestSolrDeletionPolicy1, TestSolrXmlPersistor, MoreLikeThisHandlerTest, LeaderFailoverAfterPartitionTest, SpellingQueryConverterTest, TestSolrIndexConfig, ZkSolrClientTest, BasicDistributedZkTest, SuggesterWFSTTest, ExternalCollectionsTest, TestWriterPerf, CoreAdminCreateDiscoverTest, ShardRoutingCustomTest, TestSolrJ, BlockDirectoryTest, ScriptEngineTest, TriLevelCompositeIdRoutingTest, MinimalSchemaTest, DistributedFacetPivotSmallTest, DistributedSuggestComponentTest, TestFaceting, ReturnFieldsTest, TestTrie, TestReplicationHandlerBackup, TestJmxIntegration, HighlighterMaxOffsetTest, DistributedQueryElevationComponentTest, TestStressUserVersions, TestDistribDocBasedVersion, TestQueryTypes, TestAnalyzedSuggestions, TestConfigSets, CSVRequestHandlerTest, TestRTGBase, TestModifyConfFiles, SoftAutoCommitTest, SynonymTokenizerTest, TestCloudSchemaless, IndexSchemaRuntimeFieldTest, TestWordDelimiterFilterFactory, TestValueSourceCache, ResourceLoaderTest, TestFuzzyAnalyzedSuggestions, CachingDirectoryFactoryTest, TestBinaryResponseWriter, TestMiniSolrCloudCluster, TestArbitraryIndexDir, JsonLoaderTest, HdfsLockFactoryTest, TestRecoveryHdfs, OverseerTest, BasicZkTest, TestReplicationHandler]
   [junit4] Completed on J0 in 1185.30s, 14 tests, 1 failure <<< FAILURES!

[...truncated 261 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 2 failures, 54 ignored (10 assumptions)

Total time: 199 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 469710681 bytes
Compression is 0.0%
Took 1 min 23 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 642 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/642/

1 tests failed.
REGRESSION:  org.apache.solr.handler.component.DistributedFacetPivotSmallTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=11082, name=Thread-4297, state=RUNNABLE, group=TGRP-DistributedFacetPivotSmallTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11082, name=Thread-4297, state=RUNNABLE, group=TGRP-DistributedFacetPivotSmallTest]
	at __randomizedtesting.SeedInfo.seed([3DD00CB100ACFBAF:BC3682A977F39B93]:0)
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:13614/dk_tb/bq
	at __randomizedtesting.SeedInfo.seed([3DD00CB100ACFBAF]:0)
	at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:580)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:13614/dk_tb/bq
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:580)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:575)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:152)
	at java.net.SocketInputStream.read(SocketInputStream.java:122)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:466)
	... 5 more




Build Log:
[...truncated 12549 lines...]
   [junit4] Suite: org.apache.solr.handler.component.DistributedFacetPivotSmallTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/init-core-data-001
   [junit4]   2> 2668501 T10193 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 2668502 T10193 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /dk_tb/bq
   [junit4]   2> 2668506 T10193 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2668509 T10193 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2668515 T10193 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2668528 T10193 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:13612
   [junit4]   2> 2668532 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2668532 T10193 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2668533 T10193 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr
   [junit4]   2> 2668533 T10193 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/'
   [junit4]   2> 2668578 T10193 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2668659 T10193 oasc.CoreContainer.<init> New CoreContainer 1626695483
   [junit4]   2> 2668659 T10193 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/]
   [junit4]   2> 2668660 T10193 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2668661 T10193 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2668661 T10193 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2668661 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2668662 T10193 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2668662 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2668662 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2668663 T10193 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2668663 T10193 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2668666 T10193 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2668668 T10193 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2668669 T10193 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2668669 T10193 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2668674 T10203 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2668677 T10203 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2668678 T10203 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2668678 T10203 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2668747 T10203 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2668822 T10203 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2668823 T10203 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2668832 T10203 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2669188 T10203 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2669199 T10203 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2669202 T10203 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2669211 T10203 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2669215 T10203 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2669219 T10203 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2669219 T10203 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2669220 T10203 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2669220 T10203 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2669221 T10203 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2669221 T10203 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2669221 T10203 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2669221 T10203 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2669222 T10203 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/control/data/
   [junit4]   2> 2669222 T10203 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@417763a3
   [junit4]   2> 2669223 T10203 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/control/data
   [junit4]   2> 2669223 T10203 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/control/data/index/
   [junit4]   2> 2669223 T10203 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2669224 T10203 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/control/data/index
   [junit4]   2> 2669224 T10203 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=0.23828125, floorSegmentMB=0.8798828125, forceMergeDeletesPctAllowed=16.36299440857582, segmentsPerTier=27.0, maxCFSSegmentSizeMB=1.7314453125, noCFSRatio=0.0
   [junit4]   2> 2669225 T10203 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-5cc36a0e))),segFN=segments_1,generation=1}
   [junit4]   2> 2669225 T10203 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2669225 T10203 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2669228 T10203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2669228 T10203 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2669229 T10203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2669229 T10203 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2669229 T10203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2669229 T10203 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2669230 T10203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2669230 T10203 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2669230 T10203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2669231 T10203 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2669231 T10203 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2669231 T10203 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2669232 T10203 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2669232 T10203 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2669232 T10203 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2669232 T10203 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2669233 T10203 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2669233 T10203 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2669233 T10203 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2669234 T10203 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2669234 T10203 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2669235 T10203 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2669235 T10203 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2669235 T10203 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2669236 T10203 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2669236 T10203 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2669236 T10203 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2669236 T10203 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2669237 T10203 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2669237 T10203 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2669237 T10203 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2669238 T10203 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2669238 T10203 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2669238 T10203 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2669239 T10203 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2669239 T10203 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2669240 T10203 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2669240 T10203 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2669259 T10203 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2669261 T10203 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2669263 T10203 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2669264 T10203 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2669266 T10203 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2669271 T10203 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2669271 T10203 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2669272 T10203 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=23.8154296875, floorSegmentMB=0.462890625, forceMergeDeletesPctAllowed=28.28556018731829, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5698180138024524
   [junit4]   2> 2669272 T10203 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-5cc36a0e))),segFN=segments_1,generation=1}
   [junit4]   2> 2669272 T10203 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2669273 T10203 oass.SolrIndexSearcher.<init> Opening Searcher@719c0f5b[collection1] main
   [junit4]   2> 2669273 T10203 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2669273 T10203 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2669274 T10203 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2669274 T10203 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2669274 T10203 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2669274 T10203 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2669299 T10203 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2669357 T10204 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@719c0f5b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2669360 T10203 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2669361 T10193 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 2669361 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2669366 T10193 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2669372 T10193 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2669375 T10193 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:13613
   [junit4]   2> 2669379 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2669379 T10193 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2669379 T10193 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr
   [junit4]   2> 2669380 T10193 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/'
   [junit4]   2> 2669427 T10193 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2669506 T10193 oasc.CoreContainer.<init> New CoreContainer 1123944686
   [junit4]   2> 2669507 T10193 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/]
   [junit4]   2> 2669508 T10193 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2669508 T10193 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2669509 T10193 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2669509 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2669509 T10193 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2669510 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2669510 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2669510 T10193 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2669511 T10193 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2669513 T10193 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2669516 T10193 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2669516 T10193 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2669516 T10193 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2669521 T10215 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2669524 T10215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2669524 T10215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2669525 T10215 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2669593 T10215 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2669658 T10215 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2669659 T10215 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2669668 T10215 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2670019 T10215 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2670031 T10215 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2670033 T10215 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2670042 T10215 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2670047 T10215 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2670050 T10215 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2670051 T10215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2670053 T10215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2670053 T10215 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2670054 T10215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2670054 T10215 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2670054 T10215 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2670054 T10215 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2670055 T10215 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard0/data/
   [junit4]   2> 2670055 T10215 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@417763a3
   [junit4]   2> 2670056 T10215 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard0/data
   [junit4]   2> 2670056 T10215 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard0/data/index/
   [junit4]   2> 2670056 T10215 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard0/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2670057 T10215 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard0/data/index
   [junit4]   2> 2670057 T10215 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=0.23828125, floorSegmentMB=0.8798828125, forceMergeDeletesPctAllowed=16.36299440857582, segmentsPerTier=27.0, maxCFSSegmentSizeMB=1.7314453125, noCFSRatio=0.0
   [junit4]   2> 2670058 T10215 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-64a6f0f4))),segFN=segments_1,generation=1}
   [junit4]   2> 2670058 T10215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2670059 T10215 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2670062 T10215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2670062 T10215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2670062 T10215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2670063 T10215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2670063 T10215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2670063 T10215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2670063 T10215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2670064 T10215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2670064 T10215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2670064 T10215 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2670065 T10215 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2670065 T10215 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2670065 T10215 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2670066 T10215 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2670066 T10215 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2670066 T10215 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2670067 T10215 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2670067 T10215 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2670068 T10215 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2670068 T10215 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2670069 T10215 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2670069 T10215 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2670069 T10215 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2670070 T10215 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2670070 T10215 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2670070 T10215 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2670071 T10215 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2670071 T10215 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2670071 T10215 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670072 T10215 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670072 T10215 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670073 T10215 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670073 T10215 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670074 T10215 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670074 T10215 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670074 T10215 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670075 T10215 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2670075 T10215 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2670094 T10215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2670096 T10215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2670098 T10215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2670099 T10215 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2670101 T10215 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2670106 T10215 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2670106 T10215 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2670107 T10215 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=23.8154296875, floorSegmentMB=0.462890625, forceMergeDeletesPctAllowed=28.28556018731829, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5698180138024524
   [junit4]   2> 2670107 T10215 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-64a6f0f4))),segFN=segments_1,generation=1}
   [junit4]   2> 2670107 T10215 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2670108 T10215 oass.SolrIndexSearcher.<init> Opening Searcher@2f61f507[collection1] main
   [junit4]   2> 2670108 T10215 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2670108 T10215 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2670109 T10215 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2670109 T10215 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2670109 T10215 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2670109 T10215 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2670118 T10215 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2670177 T10216 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2f61f507[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2670182 T10215 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2670183 T10193 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 2670183 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2670188 T10193 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2670194 T10193 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2670197 T10193 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:13614
   [junit4]   2> 2670201 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2670202 T10193 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2670202 T10193 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr
   [junit4]   2> 2670202 T10193 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/'
   [junit4]   2> 2670247 T10193 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2670326 T10193 oasc.CoreContainer.<init> New CoreContainer 239071772
   [junit4]   2> 2670326 T10193 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/]
   [junit4]   2> 2670327 T10193 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2670328 T10193 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2670328 T10193 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2670328 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2670329 T10193 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2670329 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2670329 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2670330 T10193 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2670330 T10193 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2670333 T10193 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2670335 T10193 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2670336 T10193 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2670336 T10193 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2670340 T10227 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2670344 T10227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2670344 T10227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2670345 T10227 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2670420 T10227 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2670479 T10227 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2670480 T10227 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2670489 T10227 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2670820 T10227 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2670830 T10227 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2670833 T10227 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2670841 T10227 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2670845 T10227 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2670848 T10227 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2670849 T10227 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2670851 T10227 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2670851 T10227 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2670852 T10227 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2670852 T10227 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2670852 T10227 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2670853 T10227 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2670853 T10227 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard1/data/
   [junit4]   2> 2670853 T10227 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@417763a3
   [junit4]   2> 2670854 T10227 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard1/data
   [junit4]   2> 2670854 T10227 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard1/data/index/
   [junit4]   2> 2670854 T10227 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2670855 T10227 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard1/data/index
   [junit4]   2> 2670855 T10227 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=0.23828125, floorSegmentMB=0.8798828125, forceMergeDeletesPctAllowed=16.36299440857582, segmentsPerTier=27.0, maxCFSSegmentSizeMB=1.7314453125, noCFSRatio=0.0
   [junit4]   2> 2670856 T10227 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-c14f4a6))),segFN=segments_1,generation=1}
   [junit4]   2> 2670856 T10227 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2670857 T10227 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2670860 T10227 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2670860 T10227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2670860 T10227 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2670860 T10227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2670861 T10227 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2670861 T10227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2670861 T10227 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2670861 T10227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2670862 T10227 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2670862 T10227 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2670862 T10227 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2670862 T10227 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2670863 T10227 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2670863 T10227 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2670864 T10227 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2670864 T10227 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2670864 T10227 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2670865 T10227 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2670865 T10227 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2670865 T10227 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2670866 T10227 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2670866 T10227 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2670867 T10227 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2670867 T10227 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2670867 T10227 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2670868 T10227 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2670868 T10227 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2670868 T10227 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2670869 T10227 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670869 T10227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670869 T10227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670870 T10227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670870 T10227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670871 T10227 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670871 T10227 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670871 T10227 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2670872 T10227 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2670872 T10227 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2670889 T10227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2670891 T10227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2670893 T10227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2670895 T10227 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2670896 T10227 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2670900 T10227 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2670901 T10227 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2670901 T10227 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=23.8154296875, floorSegmentMB=0.462890625, forceMergeDeletesPctAllowed=28.28556018731829, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5698180138024524
   [junit4]   2> 2670902 T10227 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-c14f4a6))),segFN=segments_1,generation=1}
   [junit4]   2> 2670902 T10227 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2670902 T10227 oass.SolrIndexSearcher.<init> Opening Searcher@21034156[collection1] main
   [junit4]   2> 2670903 T10227 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2670903 T10227 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2670903 T10227 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2670904 T10227 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2670904 T10227 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2670904 T10227 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2670912 T10227 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2670964 T10228 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21034156[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2670969 T10227 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2670970 T10193 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 2670970 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2670975 T10193 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2670981 T10193 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2670983 T10193 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:13616
   [junit4]   2> 2670987 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2670987 T10193 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2670988 T10193 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr
   [junit4]   2> 2670988 T10193 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/'
   [junit4]   2> 2671034 T10193 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2671116 T10193 oasc.CoreContainer.<init> New CoreContainer 279060103
   [junit4]   2> 2671116 T10193 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/]
   [junit4]   2> 2671118 T10193 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2671118 T10193 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2671118 T10193 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2671119 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2671119 T10193 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2671119 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2671120 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2671120 T10193 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2671120 T10193 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2671123 T10193 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2671125 T10193 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2671126 T10193 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2671126 T10193 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2671130 T10239 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2671133 T10239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2671134 T10239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2671134 T10239 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2671205 T10239 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2671269 T10239 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2671270 T10239 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2671280 T10239 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2671634 T10239 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2671645 T10239 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2671647 T10239 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2671656 T10239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2671660 T10239 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2671663 T10239 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2671664 T10239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2671666 T10239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2671666 T10239 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2671666 T10239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2671667 T10239 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2671667 T10239 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2671667 T10239 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2671667 T10239 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard2/data/
   [junit4]   2> 2671668 T10239 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@417763a3
   [junit4]   2> 2671668 T10239 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard2/data
   [junit4]   2> 2671669 T10239 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard2/data/index/
   [junit4]   2> 2671669 T10239 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2671669 T10239 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard2/data/index
   [junit4]   2> 2671670 T10239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=0.23828125, floorSegmentMB=0.8798828125, forceMergeDeletesPctAllowed=16.36299440857582, segmentsPerTier=27.0, maxCFSSegmentSizeMB=1.7314453125, noCFSRatio=0.0
   [junit4]   2> 2671670 T10239 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-3de52a30))),segFN=segments_1,generation=1}
   [junit4]   2> 2671671 T10239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2671671 T10239 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2671674 T10239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2671674 T10239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2671675 T10239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2671675 T10239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2671675 T10239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2671675 T10239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2671675 T10239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2671676 T10239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2671676 T10239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2671676 T10239 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2671677 T10239 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2671677 T10239 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2671677 T10239 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2671678 T10239 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2671678 T10239 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2671678 T10239 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2671679 T10239 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2671679 T10239 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2671679 T10239 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2671680 T10239 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2671680 T10239 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2671681 T10239 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2671681 T10239 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2671682 T10239 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2671682 T10239 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2671682 T10239 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2671682 T10239 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2671683 T10239 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2671683 T10239 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2671684 T10239 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2671684 T10239 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2671684 T10239 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2671685 T10239 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2671685 T10239 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2671686 T10239 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2671686 T10239 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2671686 T10239 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2671687 T10239 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2671704 T10239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2671706 T10239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2671707 T10239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2671709 T10239 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2671711 T10239 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2671715 T10239 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2671715 T10239 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2671716 T10239 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=23.8154296875, floorSegmentMB=0.462890625, forceMergeDeletesPctAllowed=28.28556018731829, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5698180138024524
   [junit4]   2> 2671717 T10239 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-3de52a30))),segFN=segments_1,generation=1}
   [junit4]   2> 2671717 T10239 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2671717 T10239 oass.SolrIndexSearcher.<init> Opening Searcher@323ba474[collection1] main
   [junit4]   2> 2671718 T10239 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2671718 T10239 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2671718 T10239 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2671718 T10239 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2671719 T10239 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2671719 T10239 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2671727 T10239 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2671784 T10240 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@323ba474[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2671789 T10239 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2671790 T10193 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 2671791 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2671795 T10193 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2671801 T10193 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2671803 T10193 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:13620
   [junit4]   2> 2671807 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2671807 T10193 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2671807 T10193 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr
   [junit4]   2> 2671808 T10193 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/'
   [junit4]   2> 2671852 T10193 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/solr.xml
   [junit4]   2> 2671942 T10193 oasc.CoreContainer.<init> New CoreContainer 607741219
   [junit4]   2> 2671942 T10193 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/]
   [junit4]   2> 2671944 T10193 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2671944 T10193 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: https
   [junit4]   2> 2671944 T10193 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2671945 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2671945 T10193 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2671945 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2671945 T10193 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2671946 T10193 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2671946 T10193 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2671949 T10193 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2671951 T10193 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2671951 T10193 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2671951 T10193 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2671955 T10251 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/'
   [junit4]   2> 2671958 T10251 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/.svn/' to classloader
   [junit4]   2> 2671959 T10251 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 2671959 T10251 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 2672024 T10251 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2672083 T10251 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2672084 T10251 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2672093 T10251 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2672421 T10251 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2672431 T10251 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2672434 T10251 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2672443 T10251 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2672446 T10251 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2672450 T10251 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2672450 T10251 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2672453 T10251 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2672453 T10251 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2672453 T10251 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2672453 T10251 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2672454 T10251 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/
   [junit4]   2> 2672454 T10251 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2672454 T10251 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard3/data/
   [junit4]   2> 2672455 T10251 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@417763a3
   [junit4]   2> 2672455 T10251 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard3/data
   [junit4]   2> 2672455 T10251 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard3/data/index/
   [junit4]   2> 2672456 T10251 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard3/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2672456 T10251 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0/temp/solr.handler.component.DistributedFacetPivotSmallTest-3DD00CB100ACFBAF-001/tempDir-001/shard3/data/index
   [junit4]   2> 2672456 T10251 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=0.23828125, floorSegmentMB=0.8798828125, forceMergeDeletesPctAllowed=16.36299440857582, segmentsPerTier=27.0, maxCFSSegmentSizeMB=1.7314453125, noCFSRatio=0.0
   [junit4]   2> 2672457 T10251 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-1ff8f51e))),segFN=segments_1,generation=1}
   [junit4]   2> 2672457 T10251 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2672458 T10251 oasc.SolrCore.initWriters created xml: solr.XMLResponseWriter
   [junit4]   2> 2672461 T10251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2672461 T10251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2672461 T10251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2672462 T10251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 2672462 T10251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2672462 T10251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2672462 T10251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2672463 T10251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 2672463 T10251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2672463 T10251 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2672463 T10251 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2672464 T10251 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2672464 T10251 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2672465 T10251 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2672465 T10251 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2672465 T10251 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2672465 T10251 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2672466 T10251 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2672466 T10251 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2672467 T10251 oasc.RequestHandlers.initHandlersFromConfig created dismax: solr.SearchHandler
   [junit4]   2> 2672467 T10251 oasc.RequestHandlers.initHandlersFromConfig created mock: org.apache.solr.core.MockQuerySenderListenerReqHandler
   [junit4]   2> 2672467 T10251 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2672468 T10251 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2672468 T10251 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2672469 T10251 oasc.RequestHandlers.initHandlersFromConfig created defaults: solr.StandardRequestHandler
   [junit4]   2> 2672469 T10251 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.StandardRequestHandler
   [junit4]   2> 2672469 T10251 oasc.RequestHandlers.initHandlersFromConfig created lazy: solr.StandardRequestHandler
   [junit4]   2> 2672469 T10251 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2672470 T10251 oasc.RequestHandlers.initHandlersFromConfig created /terms: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2672470 T10251 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2672471 T10251 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2672471 T10251 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2672471 T10251 oasc.RequestHandlers.initHandlersFromConfig created spellCheckWithWordbreak_Direct: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2672472 T10251 oasc.RequestHandlers.initHandlersFromConfig created spellCheckCompRH1: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2672472 T10251 oasc.RequestHandlers.initHandlersFromConfig created mltrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2672473 T10251 oasc.RequestHandlers.initHandlersFromConfig created tvrh: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 2672473 T10251 oasc.RequestHandlers.initHandlersFromConfig created /mlt: solr.MoreLikeThisHandler
   [junit4]   2> 2672474 T10251 oasc.RequestHandlers.initHandlersFromConfig created /debug/dump: solr.DumpRequestHandler
   [junit4]   2> 2672491 T10251 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2672492 T10251 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2672494 T10251 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2672496 T10251 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2672497 T10251 oasc.SolrCore.initDeprecatedSupport WARN solrconfig.xml uses deprecated <admin/gettableFiles>, Please update your config to use the ShowFileRequestHandler.
   [junit4]   2> 2672502 T10251 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2672502 T10251 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2672503 T10251 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=23.8154296875, floorSegmentMB=0.462890625, forceMergeDeletesPctAllowed=28.28556018731829, segmentsPerTier=13.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5698180138024524
   [junit4]   2> 2672503 T10251 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=RateLimitedDirectoryWrapper(MockDirectoryWrapper(RAMDirectory(id=lucene-1ff8f51e))),segFN=segments_1,generation=1}
   [junit4]   2> 2672503 T10251 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2672504 T10251 oass.SolrIndexSearcher.<init> Opening Searcher@7c4de4aa[collection1] main
   [junit4]   2> 2672504 T10251 oasr.RestManager.init Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 2672504 T10251 oasr.ManagedResourceStorage.load Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 2672505 T10251 oasr.ManagedResource.reloadFromStorage WARN No stored data found for /rest/managed
   [junit4]   2> 2672505 T10251 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using InMemoryStorage
   [junit4]   2> 2672505 T10251 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2672505 T10251 oashc.SpellCheckComponent.inform Initializing spell checkers
   [junit4]   2> 2672524 T10251 oass.DirectSolrSpellChecker.init init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 2672575 T10252 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c4de4aa[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2672580 T10251 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2672581 T10193 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J0
   [junit4]   2> 2672581 T10193 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> ASYNC  NEW_CORE C3865 name=collection1 org.apache.solr.core.SolrCore@1553ad14
   [junit4]   2> 2672715 T10196 C3865 oasup.LogUpdateProcessor.finish [collection1] webapp=/dk_tb/bq path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1480613377748238336)} 0 4
   [junit4]   2> ASYNC  NEW_CORE C3866 name=collection1 org.apache.solr.core.SolrCore@6702340f
   [junit4]   2> 2672845 T10208 C3866 oasup.LogUpdateProcessor.finish [collection1] webapp=/dk_tb/bq path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1480613377884553216)} 0 3
   [junit4]   2> ASYNC  NEW_CORE C3867 name=collection1 org.apache.solr.core.SolrCore@33e8cf95
   [junit4]   2> 2672976 T10220 C3867 oasup.LogUpdateProcessor.finish [collection1] webapp=/dk_tb/bq path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1480613378021916672)} 0 3
   [junit4]   2> ASYNC  NEW_CORE C3868 name=collection1 org.apache.solr.core.SolrCore@3fc4905a
   [junit4]   2> 2673107 T10232 C3868 oasup.LogUpdateProcessor.finish [collection1] webapp=/dk_tb/bq path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1480613378159280128)} 0 3
   [junit4]   2> ASYNC  NEW_CORE C3869 name=collection1 org.apache.solr.core.SolrCore@2c0a0dd2
   [junit4]   2> 2673240 T10244 C3869 oasup.LogUpdateProcessor.finish [collection1] webapp=/dk_tb/bq path=/update params={version=2&wt=javabin} {deleteByQuery=*:* (-1480613378297692160)} 0 4
   [junit4]   2> 2673248 T10197 C3865 oasup.LogUpdatePr

[...truncated too long message...]

ene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DistributedFacetPivotSmallTest -Dtests.method=testDistribSearch -Dtests.seed=3DD00CB100ACFBAF -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=zh_HK -Dtests.timezone=Asia/Jayapura -Dtests.file.encoding=UTF-8
   [junit4] ERROR    123s J0 | DistributedFacetPivotSmallTest.testDistribSearch <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11082, name=Thread-4297, state=RUNNABLE, group=TGRP-DistributedFacetPivotSmallTest]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3DD00CB100ACFBAF:BC3682A977F39B93]:0)
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:13614/dk_tb/bq
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3DD00CB100ACFBAF]:0)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:580)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:13614/dk_tb/bq
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:580)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$5.run(BaseDistributedSearchTestCase.java:575)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:152)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:122)
   [junit4]    > 	at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
   [junit4]    > 	at sun.security.ssl.InputRecord.read(InputRecord.java:480)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
   [junit4]    > 	at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:466)
   [junit4]    > 	... 5 more
   [junit4]   2> 2791074 T10193 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {id=OrdsLucene41(blocksize=128), range_facet_l=FST41, text=PostingsFormat(name=Asserting), intDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), multiDefault=FST41, _version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), timestamp=FST41, company_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), place_t=FST41}, docValues:{timestamp=DocValuesFormat(name=Lucene410)}, sim=DefaultSimilarity, locale=zh_HK, timezone=Asia/Jayapura
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=159063712,total=434634752
   [junit4]   2> NOTE: All tests run in this JVM: [TestArbitraryIndexDir, ScriptEngineTest, HdfsBasicDistributedZkTest, DistributedQueryElevationComponentTest, DistributedSpellCheckComponentTest, TestFaceting, CoreAdminRequestStatusTest, SynonymTokenizerTest, IndexSchemaRuntimeFieldTest, TestSchemaVersionResource, TestDefaultSimilarityFactory, SolrTestCaseJ4Test, DirectSolrConnectionTest, TestLeaderElectionZkExpiry, TestRangeQuery, BadComponentTest, SchemaVersionSpecificBehaviorTest, TestOrdValues, PrimUtilsTest, ExternalCollectionsTest, TermsComponentTest, QueryEqualityTest, TestValueSourceCache, TestSimpleQParserPlugin, ResourceLoaderTest, TestSchemaResource, RecoveryZkTest, HighlighterMaxOffsetTest, TestCloudSchemaless, TestCustomSort, TestAnalyzedSuggestions, BlockDirectoryTest, TestRTGBase, SoftAutoCommitTest, TriLevelCompositeIdRoutingTest, TestModifyConfFiles, TestWriterPerf, OverseerTest, TestPostingsSolrHighlighter, ConnectionManagerTest, TestReplicationHandlerBackup, TestSolrXml, TestDynamicFieldCollectionResource, TestQueryTypes, BasicDistributedZkTest, TestTrie, HdfsLockFactoryTest, MoreLikeThisHandlerTest, ZkSolrClientTest, CollectionsAPIDistributedZkTest, SyncSliceTest, TermVectorComponentDistributedTest, TestStressReorder, TestStressVersions, DistributedTermsComponentTest, SolrCmdDistributorTest, BasicFunctionalityTest, TestIndexSearcher, HighlighterTest, SimplePostToolTest, TestCoreDiscovery, TestExtendedDismaxParser, TestFoldingMultitermQuery, SuggesterTest, SuggesterWFSTTest, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, DefaultValueUpdateProcessorTest, DistanceFunctionTest, SolrInfoMBeanTest, CacheHeaderTest, TestQueryUtils, DocumentAnalysisRequestHandlerTest, DocumentBuilderTest, RegexBoostProcessorTest, TestJmxIntegration, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, TestBinaryResponseWriter, HighlighterConfigTest, UpdateParamsTest, TestSolrIndexConfig, TestStressRecovery, TestHighlightDedupGrouping, TestTolerantSearch, AliasIntegrationTest, CollectionsAPIAsyncDistributedZkTest, DeleteReplicaTest, MigrateRouteKeyTest, MultiThreadedOCPTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerStatusTest, TestCollectionAPI, TestDistribDocBasedVersion, TestRequestStatusCollectionAPI, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsSyncSliceTest, CachingDirectoryFactoryTest, TestShardHandlerFactory, TestSolrXmlPersistor, CoreAdminCreateDiscoverTest, DistributedDebugComponentTest, DistributedFacetPivotSmallTest]
   [junit4] Completed on J0 in 122.66s, 1 test, 1 error <<< FAILURES!

[...truncated 207 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 1 error, 54 ignored (10 assumptions)

Total time: 169 minutes 38 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 453743321 bytes
Compression is 0.0%
Took 1 min 18 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 641 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/641/

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12117 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/init-core-data-001
   [junit4]   2> 1189713 T7447 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1189714 T7447 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /pt/f
   [junit4]   2> 1189719 T7447 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1189720 T7447 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1189721 T7448 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1189822 T7447 oasc.ZkTestServer.run start zk server on port:47908
   [junit4]   2> 1189823 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1189824 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1189828 T7454 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@206d48d7 name:ZooKeeperConnection Watcher:127.0.0.1:47908 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1189828 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1189829 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1189829 T7447 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1189832 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1189833 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1189834 T7456 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@234d95c9 name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1189835 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1189835 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1189835 T7447 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1189837 T7447 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1189839 T7447 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1189840 T7447 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1189842 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1189842 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1189845 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1189845 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1189847 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1189848 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1189850 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1189850 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1189852 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1189852 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1189854 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1189854 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1189856 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1189857 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1189858 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1189859 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1189860 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1189861 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1189863 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1189863 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1189865 T7447 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1189865 T7447 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1189867 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1189868 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1189869 T7458 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c02faa0 name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1189869 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1189869 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 1191033 T7447 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1191047 T7447 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1191050 T7447 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:47912
   [junit4]   2> 1191052 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1191052 T7447 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1191053 T7447 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002
   [junit4]   2> 1191053 T7447 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/'
   [junit4]   2> 1191082 T7447 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/solr.xml
   [junit4]   2> 1191123 T7447 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002
   [junit4]   2> 1191123 T7447 oasc.CoreContainer.<init> New CoreContainer 1145458024
   [junit4]   2> 1191123 T7447 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/]
   [junit4]   2> 1191124 T7447 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1191125 T7447 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1191125 T7447 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1191125 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1191125 T7447 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1191125 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1191126 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1191126 T7447 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1191126 T7447 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1191128 T7447 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1191130 T7447 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1191130 T7447 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1191130 T7447 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1191131 T7447 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47908/solr
   [junit4]   2> 1191131 T7447 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1191131 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1191132 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1191134 T7469 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25084bb name:ZooKeeperConnection Watcher:127.0.0.1:47908 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1191135 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1191135 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1191137 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1191138 T7471 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@348490d5 name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1191138 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1191139 T7447 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1191141 T7447 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1191143 T7447 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1191145 T7447 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1191146 T7447 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1191148 T7447 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1191149 T7447 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47912_pt%2Ff
   [junit4]   2> 1191150 T7447 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47912_pt%2Ff
   [junit4]   2> 1191152 T7447 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1191153 T7447 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1191154 T7447 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1191155 T7447 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:47912_pt%2Ff
   [junit4]   2> 1191156 T7447 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1191157 T7447 oasc.Overseer.start Overseer (id=92531166186831876-127.0.0.1:47912_pt%2Ff-n_0000000000) starting
   [junit4]   2> 1191159 T7447 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1191164 T7447 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1191165 T7473 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1191166 T7447 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1191167 T7447 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1191168 T7447 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1191170 T7472 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1191172 T7447 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002
   [junit4]   2> 1192450 T7447 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/conf/
   [junit4]   2> 1192450 T7447 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1192451 T7447 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 1192451 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1192455 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1192456 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1192457 T7477 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e19cdf7 name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1192457 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1192457 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1192458 T7447 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1192460 T7447 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1192460 T7447 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1192461 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1192461 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1192462 T7479 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56104331 name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1192462 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1192462 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1192464 T7480 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1192466 T7472 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 1192466 T7472 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1192466 T7472 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1192468 T7472 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1192469 T7480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1192469 T7481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1193404 T7447 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1193404 T7447 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1193408 T7447 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1193410 T7447 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:47922
   [junit4]   2> 1193412 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1193412 T7447 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1193413 T7447 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003
   [junit4]   2> 1193413 T7447 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/'
   [junit4]   2> 1193442 T7447 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/solr.xml
   [junit4]   2> 1193484 T7447 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003
   [junit4]   2> 1193484 T7447 oasc.CoreContainer.<init> New CoreContainer 810882588
   [junit4]   2> 1193484 T7447 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/]
   [junit4]   2> 1193485 T7447 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1193485 T7447 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1193486 T7447 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1193486 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1193486 T7447 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1193486 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1193486 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1193487 T7447 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1193487 T7447 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1193489 T7447 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1193491 T7447 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1193491 T7447 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1193491 T7447 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1193491 T7447 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47908/solr
   [junit4]   2> 1193492 T7447 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1193492 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1193492 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1193494 T7492 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@231f37d4 name:ZooKeeperConnection Watcher:127.0.0.1:47908 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1193495 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1193495 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1193496 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1193497 T7494 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a833c3a name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1193497 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1193502 T7447 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1194506 T7447 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47922_pt%2Ff
   [junit4]   2> 1194507 T7447 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47922_pt%2Ff
   [junit4]   2> 1194510 T7447 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1194514 T7447 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003
   [junit4]   2> 1194543 T7447 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/conf/
   [junit4]   2> 1195690 T7447 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1195691 T7447 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 1195691 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1196633 T7447 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1196634 T7447 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1196638 T7447 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1196640 T7447 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:47931
   [junit4]   2> 1196642 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1196642 T7447 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1196642 T7447 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-004
   [junit4]   2> 1196643 T7447 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-004/'
   [junit4]   2> 1196672 T7447 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-004/solr.xml
   [junit4]   2> 1196714 T7447 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-004
   [junit4]   2> 1196714 T7447 oasc.CoreContainer.<init> New CoreContainer 1851904556
   [junit4]   2> 1196715 T7447 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-004/]
   [junit4]   2> 1196716 T7447 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1196716 T7447 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1196716 T7447 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1196716 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1196716 T7447 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1196717 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1196717 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1196717 T7447 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1196717 T7447 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1196719 T7447 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1196721 T7447 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1196721 T7447 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1196722 T7447 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1196722 T7447 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47908/solr
   [junit4]   2> 1196722 T7447 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1196722 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1196723 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1196724 T7507 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@202505b2 name:ZooKeeperConnection Watcher:127.0.0.1:47908 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1196725 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1196726 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1196727 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1196728 T7509 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f881d90 name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1196728 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1196732 T7447 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1197736 T7447 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47931_pt%2Ff
   [junit4]   2> 1197737 T7447 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47931_pt%2Ff
   [junit4]   2> 1197740 T7447 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1197746 T7447 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-004
   [junit4]   2> 1199009 T7447 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-004/conf/
   [junit4]   2> 1199012 T7447 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1199013 T7447 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 1199013 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1199939 T7447 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 1199940 T7447 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1199944 T7447 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1199945 T7447 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:47935
   [junit4]   2> 1199948 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1199948 T7447 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1199948 T7447 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005
   [junit4]   2> 1199948 T7447 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/'
   [junit4]   2> 1199977 T7447 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/solr.xml
   [junit4]   2> 1200019 T7447 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005
   [junit4]   2> 1200019 T7447 oasc.CoreContainer.<init> New CoreContainer 906566464
   [junit4]   2> 1200019 T7447 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/]
   [junit4]   2> 1200020 T7447 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1200021 T7447 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1200021 T7447 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1200021 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1200021 T7447 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1200021 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1200022 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1200022 T7447 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1200022 T7447 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1200024 T7447 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1200026 T7447 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1200026 T7447 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1200026 T7447 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1200026 T7447 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47908/solr
   [junit4]   2> 1200027 T7447 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1200027 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1200028 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1200029 T7522 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@472469ee name:ZooKeeperConnection Watcher:127.0.0.1:47908 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1200030 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1200030 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1200032 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1200033 T7524 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f380bdb name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1200033 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1200037 T7447 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1201041 T7447 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47935_pt%2Ff
   [junit4]   2> 1201042 T7447 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47935_pt%2Ff
   [junit4]   2> 1201045 T7447 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1201050 T7447 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005
   [junit4]   2> 1201225 T7447 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/conf/
   [junit4]   2> 1202513 T7447 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1202514 T7447 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 1202514 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1203458 T7447 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 1203458 T7447 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1203462 T7447 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1203464 T7447 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:47941
   [junit4]   2> 1203466 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1203467 T7447 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1203467 T7447 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006
   [junit4]   2> 1203467 T7447 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/'
   [junit4]   2> 1203496 T7447 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/solr.xml
   [junit4]   2> 1203538 T7447 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006
   [junit4]   2> 1203539 T7447 oasc.CoreContainer.<init> New CoreContainer 1304405461
   [junit4]   2> 1203539 T7447 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/]
   [junit4]   2> 1203540 T7447 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1203540 T7447 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1203540 T7447 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1203541 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1203541 T7447 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1203541 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1203541 T7447 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1203541 T7447 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1203542 T7447 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1203544 T7447 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1203545 T7447 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1203546 T7447 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1203546 T7447 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1203546 T7447 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47908/solr
   [junit4]   2> 1203546 T7447 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1203547 T7447 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1203547 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1203549 T7537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@760e7c17 name:ZooKeeperConnection Watcher:127.0.0.1:47908 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1203549 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1203550 T7447 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1203551 T7447 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1203552 T7539 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6299b97b name:ZooKeeperConnection Watcher:127.0.0.1:47908/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1203552 T7447 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1203556 T7447 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1204562 T7447 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47941_pt%2Ff
   [junit4]   2> 1204563 T7447 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47941_pt%2Ff
   [junit4]   2> 1204566 T7447 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1204571 T7447 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006
   [junit4]   2> 1204575 T7447 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/conf/
   [junit4]   2> 1206049 T7447 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 1206050 T7447 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 1206050 T7447 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1206184 T7484 oasha.CollectionsHandler.handleCreateAction Creating Collection : version=2&replicationFactor=2&name=solrj_collection&router.field=myOwnField&action=CREATE&wt=javabin&collection.configName=conf1&numShards=2
   [junit4]   2> 1206186 T7480 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1206186 T7473 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1206187 T7543 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 1206188 T7543 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 1206189 T7543 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 1206191 T7480 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1206193 T7472 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 1206193 T7472 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 1206193 T7472 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 1206195 T7480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206195 T7510 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206195 T7540 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206195 T7525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206195 T7481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206195 T7495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206291 T7543 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 1206292 T7543 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:47941_pt%2Ff
   [junit4]   2> 1206293 T7543 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:47922_pt%2Ff
   [junit4]   2> 1206293 T7543 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:47935_pt%2Ff
   [junit4]   2> 1206296 T7543 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:47912_pt%2Ff
   [junit4]   2> 1206416 T7486 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&shard=shard1&version=2&name=solrj_collection_shard1_replica2&collection=solrj_collection&action=CREATE&wt=javabin&collection.configName=conf1&numShards=2
   [junit4]   2> 1206420 T7529 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&shard=shard1&version=2&name=solrj_collection_shard1_replica1&collection=solrj_collection&action=CREATE&wt=javabin&collection.configName=conf1&numShards=2
   [junit4]   2> 1206420 T7517 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&shard=shard2&version=2&name=solrj_collection_shard2_replica1&collection=solrj_collection&action=CREATE&wt=javabin&collection.configName=conf1&numShards=2
   [junit4]   2> 1206420 T7486 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 1206421 T7486 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1206421 T7480 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1206422 T7529 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 1206423 T7529 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1206423 T7517 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 1206424 T7472 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:47922/pt/f",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47922_pt%2Ff",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1206424 T7517 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1206426 T7472 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:47941/pt/f",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47941_pt%2Ff",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1206427 T7480 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1206427 T7462 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&shard=shard2&version=2&name=solrj_collection_shard2_replica2&collection=solrj_collection&action=CREATE&wt=javabin&collection.configName=conf1&numShards=2
   [junit4]   2> 1206428 T7472 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:47935/pt/f",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47935_pt%2Ff",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1206430 T7462 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 1206430 T7480 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1206430 T7462 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1206431 T7472 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:47912/pt/f",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:47912_pt%2Ff",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1206533 T7540 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206533 T7525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206533 T7510 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206533 T7495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206533 T7480 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1206533 T7481 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 1207421 T7486 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 1207422 T7486 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1207423 T7486 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1207423 T7486 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1207423 T7529 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 1207424 T7486 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1207424 T7529 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1207424 T7486 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/solrj_collection_shard1_replica2/'
   [junit4]   2> 1207425 T7529 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1207425 T7529 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1207426 T7529 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1207426 T7529 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/solrj_collection_shard1_replica1/'
   [junit4]   2> 1207426 T7517 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 1207427 T7517 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1207427 T7517 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1207428 T7517 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1207428 T7517 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1207429 T7517 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/solrj_collection_shard2_replica1/'
   [junit4]   2> 1207431 T7462 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 1207432 T7462 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 1207432 T7462 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1207433 T7462 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 1207433 T7462 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1207433 T7462 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/solrj_collection_shard2_replica2/'
   [junit4]   2> 1207604 T7517 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1207605 T7486 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1207605 T7462 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1207605 T7529 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1207661 T7517 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1207661 T7486 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1207662 T7517 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1207662 T7529 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1207663 T7486 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1207663 T7462 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1207663 T7529 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1207664 T7462 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1207673 T7517 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 1207674 T7486 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 1207675 T7462 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 1207674 T7529 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 1208152 T7517 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1208153 T7462 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1208154 T7486 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1208154 T7529 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1208163 T7517 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1208164 T7529 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1208165 T7486 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1208166 T7462 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1208167 T7517 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1208167 T7529 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1208168 T7486 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1208169 T7462 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1208179 T7517 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1208180 T7529 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1208180 T7486 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1208181 T7462 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1208184 T7517 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1208185 T7529 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1208186 T7462 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1208186 T7486 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1208189 T7517 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1208190 T7529 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1208190 T7517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1208191 T7462 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1208193 T7486 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1208193 T7529 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1208194 T7529 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1208194 T7486 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1208193 T7517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1208195 T7486 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1208194 T7529 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1208194 T7462 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1208196 T7462 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1208195 T7486 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1208195 T7517 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1208196 T7462 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1208196 T7529 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1208198 T7517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1208197 T7486 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1208198 T7517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1208198 T7462 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1208198 T7529 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1208199 T7462 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1208199 T7517 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 1208199 T7486 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1208200 T7517 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1208200 T7462 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 1208200 T7529 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 1208201 T7462 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1208201 T7517 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/solrj_collection_shard2_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/solrj_collection_shard2_replica1/data/
   [junit4]   2> 1208201 T7486 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 1208203 T7517 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4a5a1d04
   [junit4]   2> 1208202 T7462 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/solrj_collection_shard2_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/solrj_collection_shard2_replica2/data/
   [junit4]   2> 1208202 T7529 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1208204 T7462 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4a5a1d04
   [junit4]   2> 1208203 T7486 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1208204 T7529 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/solrj_collection_shard1_replica1/data/
   [junit4]   2> 1208205 T7462 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/solrj_collection_shard2_replica2/data
   [junit4]   2> 1208204 T7517 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/solrj_collection_shard2_replica1/data
   [junit4]   2> 1208205 T7462 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 1208205 T7529 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4a5a1d04
   [junit4]   2> 1208205 T7486 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/solrj_collection_shard1_replica2/data/
   [junit4]   2> 1208206 T7462 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1208206 T7517 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 1208208 T7462 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-002/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 1208208 T7517 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1208207 T7529 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/solrj_collection_shard1_replica1/data
   [junit4]   2> 1208207 T7486 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4a5a1d04
   [junit4]   2> 1208209 T7529 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 1208209 T7517 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-005/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 1208210 T7486 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/solrj_collection_shard1_replica2/data
   [junit4]   2> 1208208 T7462 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=20, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47161713808175554]
   [junit4]   2> 1208211 T7486 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 1208210 T7517 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=20, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47161713808175554]
   [junit4]   2> 1208210 T7529 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1208212 T7462 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-31a77bd5)),segFN=segments_1,generation=1}
   [junit4]   2> 1208212 T7486 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1208213 T7462 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1208213 T7529 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-006/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 1208212 T7517 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-28f2f0ed)),segFN=segments_1,generation=1}
   [junit4]   2> 1208214 T7529 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=20, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47161713808175554]
   [junit4]   2> 1208213 T7486 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001/tempDir-003/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 1208214 T7517 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1208215 T7486 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=20, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.47161713808175554]
   [junit4]   2> 1208215 T7529 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-6950dfda)),segFN=segments_1,generation=1}
   [junit4]   2> 1208216 T7529 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1208216 T7486 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-305dff85)),segFN=segments_1,generation=1}
   [junit4]   2> 1208217 T7486 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1208218 T7462 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1208218 T7462 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1208219 T7462 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1208219 T7462 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1208219 T7462 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1208220 T7517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1208220 T7462 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1208220 T7517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1208221 T7462 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1208221 T7517 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1208221 T7462 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1208222 T7486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1208221 T7529 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1208221 T7517 oasu

[...truncated too long message...]

che.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 6 moreThrowable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=8316, name=Thread-5789, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:47941/pt/f, https://127.0.0.1:47935/pt/f, https://127.0.0.1:47931/pt/f, https://127.0.0.1:47922/pt/f, https://127.0.0.1:47912/pt/f]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E3B382245212DDA]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1056)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:47941/pt/f, https://127.0.0.1:47935/pt/f, https://127.0.0.1:47931/pt/f, https://127.0.0.1:47922/pt/f, https://127.0.0.1:47912/pt/f]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1632)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection3_1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 6 moreThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=8314, name=Thread-5787, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:47941/pt/f, https://127.0.0.1:47935/pt/f, https://127.0.0.1:47931/pt/f, https://127.0.0.1:47922/pt/f, https://127.0.0.1:47912/pt/f]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E3B382245212DDA]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1056)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:47941/pt/f, https://127.0.0.1:47935/pt/f, https://127.0.0.1:47931/pt/f, https://127.0.0.1:47922/pt/f, https://127.0.0.1:47912/pt/f]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1632)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection1_1
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
   [junit4]    > 	... 6 more
   [junit4]   2> 2108624 T7447 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-E3B382245212DDA-001
   [junit4]   2> 918917 T7446 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {text=PostingsFormat(name=Asserting), _version_=OrdsLucene41(blocksize=128), a_i1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), timestamp=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), intDefault=OrdsLucene41(blocksize=128), id=FSTOrd41, range_facet_l=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), other_tl1=OrdsLucene41(blocksize=128), multiDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128)))}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=ar_SA, timezone=Pacific/Rarotonga
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=135923776,total=409468928
   [junit4]   2> NOTE: All tests run in this JVM: [TestPHPSerializedResponseWriter, InfoHandlerTest, TestDocSet, HdfsDirectoryTest, RollingRestartTest, IndexBasedSpellCheckerTest, TestChildDocTransformer, FieldAnalysisRequestHandlerTest, SuggesterTSTTest, AsyncMigrateRouteKeyTest, TestImplicitCoreProperties, DocExpirationUpdateProcessorFactoryTest, SolrIndexSplitterTest, TestCoreDiscovery, TestManagedResourceStorage, DistributedQueryComponentOptimizationTest, RequestHandlersTest, SliceStateTest, UUIDFieldTest, ZkCLITest, WordBreakSolrSpellCheckerTest, TestSort, TestFileDictionaryLookup, DocValuesMultiTest, LoggingHandlerTest, SolrInfoMBeanTest, StatelessScriptUpdateProcessorFactoryTest, TestNonDefinedSimilarityFactory, BadIndexSchemaTest, SolrPluginUtilsTest, FieldMutatingUpdateProcessorTest, DeleteInactiveReplicaTest, TestSearchPerf, CustomCollectionTest, SolrXmlInZkTest, CoreAdminHandlerTest, DOMUtilTest, FileBasedSpellCheckerTest, TimeZoneUtilsTest, ReplicationFactorTest, TestReRankQParserPlugin, TermVectorComponentDistributedTest, SearchHandlerTest, DistributedIntervalFacetingTest, TestFastLRUCache, UniqFieldsUpdateProcessorFactoryTest, HdfsRecoveryZkTest, TestCSVLoader, TestSchemaNameResource, TestFastWriter, TestInfoStreamLogging, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J3 in 919.71s, 1 test, 1 error <<< FAILURES!

[...truncated 659 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 1 error, 54 ignored (10 assumptions)

Total time: 176 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 683147863 bytes
Compression is 0.0%
Took 2 min 2 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 640 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/640/

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11568 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/init-core-data-001
   [junit4]   2> 90181 T162 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 90181 T162 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 90190 T162 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 90191 T162 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 90192 T163 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 90292 T162 oasc.ZkTestServer.run start zk server on port:22182
   [junit4]   2> 90293 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 90294 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 90298 T169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a967515 name:ZooKeeperConnection Watcher:127.0.0.1:22182 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 90299 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 90299 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 90300 T162 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 90303 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 90304 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 90306 T171 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2796bcfd name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 90307 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 90307 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 90308 T162 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 90312 T162 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 90316 T162 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 90319 T162 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 90322 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 90324 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 90329 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 90330 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 90335 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 90336 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 90340 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 90340 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 90344 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 90345 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 90349 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 90350 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 90354 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 90355 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 90359 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 90360 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 90363 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 90364 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 90368 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 90369 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 90373 T162 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 90374 T162 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 90385 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 90386 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 90388 T173 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5669be6f name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 90388 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 90389 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 91754 T162 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 91827 T162 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 91833 T162 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:22186
   [junit4]   2> 91836 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 91837 T162 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 91837 T162 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-002
   [junit4]   2> 91837 T162 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-002/'
   [junit4]   2> 91869 T162 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-002/solr.xml
   [junit4]   2> 91944 T162 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-002
   [junit4]   2> 91945 T162 oasc.CoreContainer.<init> New CoreContainer 168861149
   [junit4]   2> 91945 T162 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-002/]
   [junit4]   2> 91947 T162 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 91947 T162 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 91947 T162 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 91948 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 91948 T162 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 91948 T162 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 91949 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 91949 T162 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 91949 T162 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 91952 T162 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 91955 T162 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 91955 T162 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 91956 T162 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 91956 T162 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22182/solr
   [junit4]   2> 91956 T162 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 91957 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 91958 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 91960 T184 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d8f92da name:ZooKeeperConnection Watcher:127.0.0.1:22182 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 91960 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 91961 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 91963 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 91965 T186 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@491f7520 name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 91965 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 91967 T162 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 91970 T162 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 91973 T162 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 91976 T162 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 91979 T162 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 91983 T162 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 91985 T162 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22186_
   [junit4]   2> 91986 T162 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22186_
   [junit4]   2> 91991 T162 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 91993 T162 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 91995 T162 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 91997 T162 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:22186_
   [junit4]   2> 91998 T162 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 92001 T162 oasc.Overseer.start Overseer (id=92524077304709124-127.0.0.1:22186_-n_0000000000) starting
   [junit4]   2> 92004 T162 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 92017 T162 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 92018 T188 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 92019 T162 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 92022 T162 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 92024 T162 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 92029 T187 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 92032 T162 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-002
   [junit4]   2> 92190 T162 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-002/conf/
   [junit4]   2> 93457 T162 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 93459 T162 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 93459 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 93488 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 93489 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 93490 T192 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bc81b5e name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 93490 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 93491 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 93492 T162 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 93497 T162 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 93498 T162 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 93498 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 93499 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 93500 T194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51754158 name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 93500 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 93501 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 93503 T195 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 93507 T187 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 93508 T187 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 93517 T187 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 93522 T187 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 93524 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 93524 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 94554 T162 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 94555 T162 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 94562 T162 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 94565 T162 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:22191
   [junit4]   2> 94568 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 94568 T162 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 94568 T162 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003
   [junit4]   2> 94569 T162 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/'
   [junit4]   2> 94602 T162 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/solr.xml
   [junit4]   2> 94654 T162 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003
   [junit4]   2> 94655 T162 oasc.CoreContainer.<init> New CoreContainer 396200443
   [junit4]   2> 94655 T162 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/]
   [junit4]   2> 94656 T162 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 94656 T162 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 94657 T162 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 94657 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 94657 T162 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 94658 T162 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 94658 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 94658 T162 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 94658 T162 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 94662 T162 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 94664 T162 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 94665 T162 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 94665 T162 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 94665 T162 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22182/solr
   [junit4]   2> 94666 T162 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 94666 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 94667 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 94669 T207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64790ec1 name:ZooKeeperConnection Watcher:127.0.0.1:22182 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 94669 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 94670 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 94673 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 94674 T209 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70533bef name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 94675 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 94682 T162 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 95688 T162 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22191_
   [junit4]   2> 95690 T162 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22191_
   [junit4]   2> 95694 T162 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 95699 T162 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003
   [junit4]   2> 96741 T162 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/conf/
   [junit4]   2> 96824 T162 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 96826 T162 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 96826 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 97841 T162 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 97842 T162 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 97848 T162 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 97851 T162 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:22194
   [junit4]   2> 97855 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 97855 T162 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 97856 T162 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004
   [junit4]   2> 97856 T162 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/'
   [junit4]   2> 97887 T162 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/solr.xml
   [junit4]   2> 97933 T162 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004
   [junit4]   2> 97934 T162 oasc.CoreContainer.<init> New CoreContainer 764430233
   [junit4]   2> 97934 T162 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/]
   [junit4]   2> 97935 T162 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 97935 T162 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 97936 T162 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 97936 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 97936 T162 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 97937 T162 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 97937 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 97937 T162 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 97937 T162 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 97940 T162 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 97943 T162 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 97943 T162 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 97944 T162 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 97944 T162 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22182/solr
   [junit4]   2> 97944 T162 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 97945 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 97945 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 97947 T222 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37021349 name:ZooKeeperConnection Watcher:127.0.0.1:22182 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 97948 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 97948 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 97950 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 97952 T224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23b8655f name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 97952 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 97960 T162 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 98968 T162 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22194_
   [junit4]   2> 98970 T162 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22194_
   [junit4]   2> 98975 T162 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 98980 T162 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004
   [junit4]   2> 98984 T162 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/conf/
   [junit4]   2> 99920 T162 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 99921 T162 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 99922 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 101052 T162 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 101053 T162 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 101060 T162 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 101063 T162 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:25361
   [junit4]   2> 101067 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 101067 T162 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 101067 T162 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005
   [junit4]   2> 101068 T162 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/'
   [junit4]   2> 101102 T162 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/solr.xml
   [junit4]   2> 101153 T162 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005
   [junit4]   2> 101153 T162 oasc.CoreContainer.<init> New CoreContainer 1944647959
   [junit4]   2> 101154 T162 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/]
   [junit4]   2> 101155 T162 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 101155 T162 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 101156 T162 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 101156 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 101156 T162 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 101157 T162 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 101157 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 101157 T162 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 101158 T162 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 101161 T162 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 101163 T162 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 101164 T162 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 101164 T162 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 101165 T162 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22182/solr
   [junit4]   2> 101165 T162 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 101165 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 101166 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 101168 T237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d85b85e name:ZooKeeperConnection Watcher:127.0.0.1:22182 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 101169 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 101170 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 101172 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 101174 T239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58367fb6 name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 101175 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 101184 T162 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 102198 T162 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:25361_
   [junit4]   2> 102200 T162 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:25361_
   [junit4]   2> 102206 T162 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 102212 T162 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005
   [junit4]   2> 102217 T162 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/conf/
   [junit4]   2> 103372 T162 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 103373 T162 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 103374 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 104499 T162 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 104500 T162 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 104506 T162 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 104509 T162 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:51855
   [junit4]   2> 104512 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 104512 T162 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 104512 T162 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006
   [junit4]   2> 104513 T162 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/'
   [junit4]   2> 104544 T162 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/solr.xml
   [junit4]   2> 104599 T162 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006
   [junit4]   2> 104599 T162 oasc.CoreContainer.<init> New CoreContainer 213760074
   [junit4]   2> 104600 T162 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/]
   [junit4]   2> 104601 T162 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 104601 T162 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 104601 T162 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 104601 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 104602 T162 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 104602 T162 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 104602 T162 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 104603 T162 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 104603 T162 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 104606 T162 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 104608 T162 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 104609 T162 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 104609 T162 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 104610 T162 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:22182/solr
   [junit4]   2> 104610 T162 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 104610 T162 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 104611 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 104613 T252 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c083860 name:ZooKeeperConnection Watcher:127.0.0.1:22182 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 104614 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 104614 T162 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 104617 T162 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 104618 T254 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51b39791 name:ZooKeeperConnection Watcher:127.0.0.1:22182/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 104619 T162 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 104626 T162 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 105633 T162 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51855_
   [junit4]   2> 105635 T162 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51855_
   [junit4]   2> 105641 T162 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 105646 T162 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006
   [junit4]   2> 106347 T162 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/conf/
   [junit4]   2> 106358 T162 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 106360 T162 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 106360 T162 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 106624 T202 oasha.CollectionsHandler.handleCreateAction Creating Collection : wt=javabin&replicationFactor=2&collection.configName=conf1&version=2&router.field=myOwnField&name=solrj_collection&action=CREATE&numShards=2
   [junit4]   2> 106627 T195 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106628 T188 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 106634 T258 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 106641 T258 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 106642 T258 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 106647 T195 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106652 T187 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 106652 T187 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 106652 T187 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 106656 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106656 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106656 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106656 T240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106656 T255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106656 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106748 T258 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 106748 T258 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:51855_
   [junit4]   2> 106761 T258 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:22191_
   [junit4]   2> 106764 T258 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:25361_
   [junit4]   2> 106767 T258 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:22194_
   [junit4]   2> 106930 T229 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&collection=solrj_collection&shard=shard2&collection.configName=conf1&qt=/admin/cores&version=2&name=solrj_collection_shard2_replica1&action=CREATE&numShards=2
   [junit4]   2> 106931 T244 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&collection=solrj_collection&shard=shard1&collection.configName=conf1&qt=/admin/cores&version=2&name=solrj_collection_shard1_replica1&action=CREATE&numShards=2
   [junit4]   2> 106933 T204 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&collection=solrj_collection&shard=shard1&collection.configName=conf1&qt=/admin/cores&version=2&name=solrj_collection_shard1_replica2&action=CREATE&numShards=2
   [junit4]   2> 106935 T217 oasha.CoreAdminHandler.handleCreateAction core create command wt=javabin&collection=solrj_collection&shard=shard2&collection.configName=conf1&qt=/admin/cores&version=2&name=solrj_collection_shard2_replica2&action=CREATE&numShards=2
   [junit4]   2> 106946 T244 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 106947 T229 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 106946 T217 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 106946 T204 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 106949 T229 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 106948 T244 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 106948 T195 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106950 T204 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 106949 T217 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 106955 T187 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:51855_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:51855",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 106960 T187 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:25361_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:25361",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 106963 T195 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106965 T187 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:22194_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:22194",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 106969 T187 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "node_name":"127.0.0.1:22191_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:22191",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 106972 T195 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 106974 T255 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106974 T240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106974 T196 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106974 T195 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106974 T210 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 106974 T225 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 107950 T229 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 107950 T244 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 107951 T229 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 107951 T204 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 107951 T244 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 107952 T229 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 107952 T204 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 107953 T244 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 107953 T229 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 107983 T204 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 107983 T244 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 107985 T229 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 107983 T217 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 107985 T244 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 107985 T217 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 107985 T229 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/solrj_collection_shard2_replica1/'
   [junit4]   2> 107984 T204 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 107987 T217 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 107986 T244 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/solrj_collection_shard1_replica1/'
   [junit4]   2> 107988 T204 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 107988 T217 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 107989 T204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/solrj_collection_shard1_replica2/'
   [junit4]   2> 107990 T217 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 107990 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/solrj_collection_shard2_replica2/'
   [junit4]   2> 108125 T204 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 108126 T244 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 108127 T217 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 108127 T229 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 108199 T244 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 108200 T204 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 108200 T244 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 108201 T229 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 108202 T204 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 108203 T229 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 108204 T217 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 108206 T217 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 108215 T244 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 108220 T204 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 108221 T217 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 108222 T229 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 108790 T204 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 108805 T204 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 108808 T204 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 108827 T217 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 108828 T204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108833 T204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108834 T229 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 108835 T217 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 108838 T217 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 108838 T204 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108839 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108842 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108842 T204 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108844 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108844 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108845 T204 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 108845 T204 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 108845 T229 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 108846 T204 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/solrj_collection_shard1_replica2/data/
   [junit4]   2> 108848 T204 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@735ea310
   [junit4]   2> 108850 T204 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/solrj_collection_shard1_replica2/data
   [junit4]   2> 108850 T204 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 108851 T229 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 108851 T204 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 108852 T204 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-003/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 108853 T204 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=8, maxMergedSegmentMB=89.6044921875, floorSegmentMB=0.3974609375, forceMergeDeletesPctAllowed=11.74795790304424, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7073315922145454
   [junit4]   2> 108855 T204 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-310f0653)),segFN=segments_1,generation=1}
   [junit4]   2> 108855 T204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 108863 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 108863 T217 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108868 T244 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 108870 T217 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108874 T229 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108876 T244 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 108877 T217 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108878 T217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108879 T217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108879 T244 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 108879 T217 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108881 T217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108881 T229 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108881 T217 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108882 T217 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 108883 T217 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 108883 T217 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/solrj_collection_shard2_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/solrj_collection_shard2_replica2/data/
   [junit4]   2> 108884 T217 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@735ea310
   [junit4]   2> 108885 T217 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/solrj_collection_shard2_replica2/data
   [junit4]   2> 108885 T217 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 108886 T217 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/solrj_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 108886 T217 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-004/solrj_collection_shard2_replica2/data/index
   [junit4]   2> 108887 T217 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=8, maxMergedSegmentMB=89.6044921875, floorSegmentMB=0.3974609375, forceMergeDeletesPctAllowed=11.74795790304424, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7073315922145454
   [junit4]   2> 108887 T229 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108888 T217 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-59f88e8c)),segFN=segments_1,generation=1}
   [junit4]   2> 108889 T217 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 108889 T229 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108889 T229 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108890 T229 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108891 T229 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108892 T229 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108892 T229 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 108893 T229 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 108894 T229 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at ../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/solrj_collection_shard2_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/solrj_collection_shard2_replica1/data/
   [junit4]   2> 108894 T229 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@735ea310
   [junit4]   2> 108894 T217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 108896 T229 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/solrj_collection_shard2_replica1/data
   [junit4]   2> 108896 T229 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 108897 T229 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 108898 T229 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/../../../../../../../../../../../usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-005/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 108899 T229 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=8, maxMergedSegmentMB=89.6044921875, floorSegmentMB=0.3974609375, forceMergeDeletesPctAllowed=11.74795790304424, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7073315922145454
   [junit4]   2> 108900 T244 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108900 T229 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-2bc56e1f)),segFN=segments_1,generation=1}
   [junit4]   2> 108901 T229 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 108904 T244 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 108909 T229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 108909 T244 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108911 T244 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108911 T244 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108911 T244 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 108912 T244 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 108913 T244 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 108913 T244 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 108914 T244 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 108914 T244 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/solrj_collection_shard1_replica1/data/
   [junit4]   2> 108915 T244 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@735ea310
   [junit4]   2> 108916 T244 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/solrj_collection_shard1_replica1/data
   [junit4]   2> 108916 T244 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 108917 T244 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 108917 T244 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001/tempDir-006/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 108918 T244 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=8, maxMergedSegmentMB=89.6044921875, floorSegmentMB=0.3974609375, forceMergeDeletesPctAllowed=11.74795790304424, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7073315922145454
   [junit4]   2> 108920 T244 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-2d0f18ad)),segFN=segments_1,generation=1}
   [junit4]   2> 108920 T244 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 108926 T244 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 108930 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 108930 T229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 108930 T217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 108932 T229 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 108931 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 108932 T229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108932 T217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 108933 T229 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108934 T244 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 108933 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108935 T244 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 108934 T229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 108934 T217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108936 T229 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108935 T244 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108935 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108937 T244 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108937 T229 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108936 T217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 108938 T229 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 108938 T244 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 108938 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 108940 T244 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108940 T204 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108940 T244 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108940 T229 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108939 T217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 108941 T244 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 108941 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108942 T217 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108942 T229 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108943 T217 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 108943 T244 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108943 T204 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 108944 T229 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108945 T204 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108944 T217 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 108946 T204 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108946 T229 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 108945 T244 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108947 T204 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108947 T217 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108948 T204 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 108948 T217 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108948 T204 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 108947 T244 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 108947 T229 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 108949 T244 oasc.RequestHandlers.initHandlersFromConfig cr

[...truncated too long message...]

.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=625, name=Thread-215, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection5_5
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([72D7C5C8CEE77426]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=622, name=Thread-212, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection2_5
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([72D7C5C8CEE77426]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=624, name=Thread-214, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection4_5
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([72D7C5C8CEE77426]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=621, name=Thread-211, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection1_5
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([72D7C5C8CEE77426]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=623, name=Thread-213, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection3_5
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([72D7C5C8CEE77426]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=620, name=Thread-210, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not fully create collection: awholynewstresscollection_collection0_5
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([72D7C5C8CEE77426]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1627)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1653)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:1037)
   [junit4]   2> 896666 T162 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-72D7C5C8CEE77426-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=cs_CZ, timezone=UTC
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=19090280,total=361234432
   [junit4]   2> NOTE: All tests run in this JVM: [DistributedQueryComponentOptimizationTest, TestMergePolicyConfig, ClusterStateUpdateTest, SpellCheckComponentTest, DocumentAnalysisRequestHandlerTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J3 in 806.55s, 1 test, 1 error <<< FAILURES!

[...truncated 1163 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 1 error, 54 ignored (10 assumptions)

Total time: 200 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 659715220 bytes
Compression is 0.0%
Took 1 min 54 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 639 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/639/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testDistribSearch

Error Message:
Error executing query

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Error executing query
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:100)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:223)
	at org.apache.solr.cloud.CloudInspectUtil.compareResults(CloudInspectUtil.java:165)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrServer(FullSolrCloudDistribCmdsTest.java:414)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.doTest(FullSolrCloudDistribCmdsTest.java:144)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.ParseException: Invalid content type: 
	at org.apache.http.entity.ContentType.parse(ContentType.java:233)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:502)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:212)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:208)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
	... 46 more


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Error CREATEing SolrCore 'halfcollection_shard1_replica1': Unable to create core [halfcollection_shard1_replica1] Caused by: Could not get shard id for core: halfcollection_shard1_replica1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:212)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:208)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:583)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:205)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11830 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/init-core-data-001
   [junit4]   2> 346782 T1037 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 346782 T1037 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 346795 T1037 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 346797 T1037 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 346798 T1038 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 346898 T1037 oasc.ZkTestServer.run start zk server on port:24069
   [junit4]   2> 346899 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 346901 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 346906 T1044 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43e18e5a name:ZooKeeperConnection Watcher:127.0.0.1:24069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 346907 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 346907 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 346908 T1037 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 346912 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 346914 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 346915 T1046 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@792ab607 name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 346916 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 346916 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 346917 T1037 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 346919 T1037 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 346921 T1037 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 346923 T1037 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 346926 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 346927 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 346931 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 346932 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 346935 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 346936 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 346939 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 346940 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 346942 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 346943 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 346946 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 346947 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 346950 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 346950 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 346953 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 346954 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 346957 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 346958 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 346961 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 346962 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 346964 T1037 oasc.AbstractZkTestCase.putConfig put /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 346966 T1037 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 346968 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 346969 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 346971 T1048 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@394b5c1e name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 346972 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 346972 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> Using new style solr.xml
   [junit4]   2> 349012 T1037 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 349020 T1037 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 349029 T1037 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:26594
   [junit4]   2> 349035 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 349036 T1037 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 349036 T1037 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002
   [junit4]   2> 349036 T1037 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/'
   [junit4]   2> 349114 T1037 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/solr.xml
   [junit4]   2> 349208 T1037 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002
   [junit4]   2> 349209 T1037 oasc.CoreContainer.<init> New CoreContainer 1996485309
   [junit4]   2> 349209 T1037 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/]
   [junit4]   2> 349211 T1037 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 349211 T1037 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 349212 T1037 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 349212 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 349213 T1037 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 349213 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 349214 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 349214 T1037 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 349215 T1037 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 349220 T1037 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 349225 T1037 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 349226 T1037 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 349226 T1037 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 349227 T1037 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24069/solr
   [junit4]   2> 349228 T1037 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 349228 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 349229 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 349231 T1059 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aca4765 name:ZooKeeperConnection Watcher:127.0.0.1:24069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 349232 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 349233 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 349235 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 349237 T1061 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56979ff0 name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 349237 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 349239 T1037 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 349242 T1037 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 349245 T1037 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 349248 T1037 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 349251 T1037 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 349255 T1037 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 349257 T1037 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:26594_
   [junit4]   2> 349258 T1037 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:26594_
   [junit4]   2> 349261 T1037 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 349263 T1037 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 349265 T1037 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 349268 T1037 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:26594_
   [junit4]   2> 349268 T1037 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 349270 T1037 oasc.Overseer.start Overseer (id=92518105655541764-127.0.0.1:26594_-n_0000000000) starting
   [junit4]   2> 349274 T1037 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 349283 T1037 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 349285 T1063 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 349286 T1037 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 349290 T1037 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 349291 T1037 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 349295 T1062 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 349299 T1037 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002
   [junit4]   2> 350912 T1037 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/conf/
   [junit4]   2> 350917 T1037 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 350918 T1037 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 350919 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 350928 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 350929 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 350931 T1067 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d45b726 name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 350931 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 350932 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 350933 T1037 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 350936 T1037 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 350937 T1037 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 350938 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 350942 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 350946 T1069 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@60e6cfb3 name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 350946 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 350946 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 350948 T1070 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 350951 T1062 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 350951 T1062 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 350952 T1062 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 350954 T1062 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 350956 T1070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 350969 T1071 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 352979 T1037 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 352980 T1037 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 352991 T1037 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 352996 T1037 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:31688
   [junit4]   2> 353001 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 353002 T1037 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 353003 T1037 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003
   [junit4]   2> 353003 T1037 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/'
   [junit4]   2> 353070 T1037 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/solr.xml
   [junit4]   2> 353164 T1037 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003
   [junit4]   2> 353165 T1037 oasc.CoreContainer.<init> New CoreContainer 256180486
   [junit4]   2> 353165 T1037 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/]
   [junit4]   2> 353167 T1037 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 353168 T1037 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 353168 T1037 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 353169 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 353169 T1037 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 353170 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 353170 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 353171 T1037 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 353172 T1037 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 353176 T1037 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 353180 T1037 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 353180 T1037 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 353181 T1037 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 353181 T1037 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24069/solr
   [junit4]   2> 353182 T1037 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 353182 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 353183 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 353185 T1082 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@670e65e5 name:ZooKeeperConnection Watcher:127.0.0.1:24069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 353186 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 353187 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 353189 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 353191 T1084 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9da2dc8 name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 353191 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 353198 T1037 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 354204 T1037 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31688_
   [junit4]   2> 354205 T1037 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31688_
   [junit4]   2> 354211 T1037 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 354218 T1037 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003
   [junit4]   2> 355721 T1037 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/conf/
   [junit4]   2> 355722 T1037 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 355723 T1037 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 355724 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 357776 T1037 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 357781 T1037 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 357796 T1037 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 357801 T1037 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:37852
   [junit4]   2> 357807 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 357808 T1037 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 357809 T1037 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004
   [junit4]   2> 357809 T1037 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/'
   [junit4]   2> 357877 T1037 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/solr.xml
   [junit4]   2> 357969 T1037 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004
   [junit4]   2> 357970 T1037 oasc.CoreContainer.<init> New CoreContainer 1877643730
   [junit4]   2> 357970 T1037 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/]
   [junit4]   2> 357972 T1037 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 357973 T1037 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 357973 T1037 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 357974 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 357974 T1037 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 357975 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 357975 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 357976 T1037 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 357976 T1037 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 357981 T1037 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 357985 T1037 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 357986 T1037 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 357986 T1037 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 357987 T1037 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24069/solr
   [junit4]   2> 357987 T1037 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 357987 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 357989 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 357990 T1097 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@702d405a name:ZooKeeperConnection Watcher:127.0.0.1:24069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 357992 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 357993 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 357995 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 357996 T1099 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4849e41e name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 357997 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 358004 T1037 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 359016 T1037 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37852_
   [junit4]   2> 359017 T1037 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37852_
   [junit4]   2> 359022 T1037 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 359032 T1037 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004
   [junit4]   2> 360345 T1037 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/conf/
   [junit4]   2> 360590 T1037 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 360591 T1037 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 360592 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 362694 T1037 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 362698 T1037 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 362706 T1037 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 362709 T1037 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:44211
   [junit4]   2> 362715 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 362716 T1037 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 362716 T1037 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005
   [junit4]   2> 362717 T1037 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/'
   [junit4]   2> 362791 T1037 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/solr.xml
   [junit4]   2> 362880 T1037 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005
   [junit4]   2> 362881 T1037 oasc.CoreContainer.<init> New CoreContainer 78079296
   [junit4]   2> 362882 T1037 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/]
   [junit4]   2> 362883 T1037 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 362883 T1037 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 362884 T1037 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 362884 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 362884 T1037 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 362885 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 362885 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 362886 T1037 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 362886 T1037 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 362891 T1037 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 362895 T1037 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 362896 T1037 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 362897 T1037 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 362897 T1037 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24069/solr
   [junit4]   2> 362898 T1037 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 362898 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 362899 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 362901 T1112 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a1ce70a name:ZooKeeperConnection Watcher:127.0.0.1:24069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 362902 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 362903 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 362905 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 362907 T1114 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@68e52e6 name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 362907 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 362914 T1037 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 363920 T1037 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44211_
   [junit4]   2> 363921 T1037 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44211_
   [junit4]   2> 363926 T1037 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 363932 T1037 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005
   [junit4]   2> 365218 T1037 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/conf/
   [junit4]   2> 365397 T1037 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 365399 T1037 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 365399 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 367565 T1037 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 367566 T1037 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 367578 T1037 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 367583 T1037 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:50737
   [junit4]   2> 367589 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 367589 T1037 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 367590 T1037 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-006
   [junit4]   2> 367590 T1037 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-006/'
   [junit4]   2> 367657 T1037 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-006/solr.xml
   [junit4]   2> 367751 T1037 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-006
   [junit4]   2> 367752 T1037 oasc.CoreContainer.<init> New CoreContainer 322846844
   [junit4]   2> 367752 T1037 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-006/]
   [junit4]   2> 367754 T1037 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 367755 T1037 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 367755 T1037 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 367756 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 367756 T1037 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 367757 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 367757 T1037 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 367758 T1037 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 367758 T1037 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 367764 T1037 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 367769 T1037 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 367769 T1037 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 367770 T1037 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 367770 T1037 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:24069/solr
   [junit4]   2> 367771 T1037 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 367771 T1037 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 367772 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 367774 T1127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17b1eea5 name:ZooKeeperConnection Watcher:127.0.0.1:24069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 367776 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 367777 T1037 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 367779 T1037 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 367781 T1129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63195110 name:ZooKeeperConnection Watcher:127.0.0.1:24069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 367781 T1037 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 367788 T1037 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 368794 T1037 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50737_
   [junit4]   2> 368795 T1037 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50737_
   [junit4]   2> 368800 T1037 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 368807 T1037 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-006
   [junit4]   2> 370108 T1037 oasc.CorePropertiesLocator.discoverUnder Found core conf in /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-006/conf/
   [junit4]   2> 370206 T1037 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 370207 T1037 oass.SolrDispatchFilter.init user.dir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3
   [junit4]   2> 370208 T1037 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 370387 T1077 oasha.CollectionsHandler.handleCreateAction Creating Collection : version=2&wt=javabin&replicationFactor=2&router.field=myOwnField&numShards=2&name=solrj_collection&action=CREATE&collection.configName=conf1
   [junit4]   2> 370389 T1070 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370390 T1063 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 370392 T1133 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"solrj_collection",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "collection.configName":"conf1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "router.field":"myOwnField"}
   [junit4]   2> 370394 T1133 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/solrj_collection 
   [junit4]   2> 370395 T1133 oascc.SolrZkClient.makePath makePath: /collections/solrj_collection
   [junit4]   2> 370398 T1070 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370402 T1062 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: solrj_collection
   [junit4]   2> 370402 T1062 oasc.Overseer$ClusterStateUpdater.createCollection Create collection solrj_collection with shards [shard1, shard2]
   [junit4]   2> 370402 T1062 oasc.Overseer$ClusterStateUpdater.createCollection state version solrj_collection 1
   [junit4]   2> 370405 T1071 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370405 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370405 T1130 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370405 T1115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370405 T1070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370405 T1085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370499 T1133 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection solrj_collection, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 370500 T1133 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica1 as part of slice shard1 of collection solrj_collection on 127.0.0.1:26594_
   [junit4]   2> 370501 T1133 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard1_replica2 as part of slice shard1 of collection solrj_collection on 127.0.0.1:37852_
   [junit4]   2> 370504 T1133 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica1 as part of slice shard2 of collection solrj_collection on 127.0.0.1:31688_
   [junit4]   2> 370506 T1133 oasc.OverseerCollectionProcessor.createCollection Creating shard solrj_collection_shard2_replica2 as part of slice shard2 of collection solrj_collection on 127.0.0.1:44211_
   [junit4]   2> 370663 T1105 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=solrj_collection&wt=javabin&qt=/admin/cores&shard=shard2&numShards=2&name=solrj_collection_shard2_replica2&action=CREATE&collection.configName=conf1
   [junit4]   2> 370666 T1091 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=solrj_collection&wt=javabin&qt=/admin/cores&shard=shard1&numShards=2&name=solrj_collection_shard1_replica2&action=CREATE&collection.configName=conf1
   [junit4]   2> 370666 T1075 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=solrj_collection&wt=javabin&qt=/admin/cores&shard=shard2&numShards=2&name=solrj_collection_shard2_replica1&action=CREATE&collection.configName=conf1
   [junit4]   2> 370668 T1052 oasha.CoreAdminHandler.handleCreateAction core create command version=2&collection=solrj_collection&wt=javabin&qt=/admin/cores&shard=shard1&numShards=2&name=solrj_collection_shard1_replica1&action=CREATE&collection.configName=conf1
   [junit4]   2> 370670 T1105 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica2 state=down collection=solrj_collection
   [junit4]   2> 370670 T1091 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica2 state=down collection=solrj_collection
   [junit4]   2> 370671 T1075 oasc.ZkController.publish publishing core=solrj_collection_shard2_replica1 state=down collection=solrj_collection
   [junit4]   2> 370671 T1105 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 370672 T1091 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 370671 T1070 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370672 T1052 oasc.ZkController.publish publishing core=solrj_collection_shard1_replica1 state=down collection=solrj_collection
   [junit4]   2> 370672 T1075 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 370674 T1052 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 370676 T1062 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "node_name":"127.0.0.1:44211_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:44211",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica2"}
   [junit4]   2> 370679 T1062 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "node_name":"127.0.0.1:37852_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:37852",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica2"}
   [junit4]   2> 370682 T1070 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370683 T1062 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "node_name":"127.0.0.1:31688_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:31688",
   [junit4]   2> 	  "core":"solrj_collection_shard2_replica1"}
   [junit4]   2> 370686 T1062 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "node_name":"127.0.0.1:26594_",
   [junit4]   2> 	  "collection":"solrj_collection",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:26594",
   [junit4]   2> 	  "core":"solrj_collection_shard1_replica1"}
   [junit4]   2> 370688 T1070 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 370689 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370690 T1071 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370690 T1085 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370689 T1070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370689 T1130 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 370689 T1115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 371672 T1105 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica2
   [junit4]   2> 371673 T1105 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 371673 T1091 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica2
   [junit4]   2> 371674 T1075 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard2_replica1
   [junit4]   2> 371675 T1091 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 371675 T1052 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for solrj_collection_shard1_replica1
   [junit4]   2> 371675 T1075 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 371677 T1091 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 371677 T1075 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 371678 T1075 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 371676 T1052 oasc.ZkController.createCollectionZkNode Check for collection zkNode:solrj_collection
   [junit4]   2> 371676 T1105 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 371680 T1052 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 371677 T1091 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 371681 T1052 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 371680 T1105 oascc.ZkStateReader.readConfigName Load collection config from:/collections/solrj_collection
   [junit4]   2> 371680 T1075 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 371683 T1091 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 371683 T1105 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 371684 T1052 oascc.ZkStateReader.readConfigName path=/collections/solrj_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 371683 T1091 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/solrj_collection_shard1_replica2/'
   [junit4]   2> 371683 T1075 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/solrj_collection_shard2_replica1/'
   [junit4]   2> 371685 T1052 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/solrj_collection_shard1_replica1/'
   [junit4]   2> 371684 T1105 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/solrj_collection_shard2_replica2/'
   [junit4]   2> 372061 T1052 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 372062 T1091 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 372063 T1075 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 372070 T1105 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 372169 T1052 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 372202 T1075 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 372203 T1052 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 372205 T1075 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 372212 T1105 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 372213 T1091 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 372216 T1105 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 372221 T1052 oass.IndexSchema.readSchema [solrj_collection_shard1_replica1] Schema name=test
   [junit4]   2> 372230 T1105 oass.IndexSchema.readSchema [solrj_collection_shard2_replica2] Schema name=test
   [junit4]   2> 372232 T1075 oass.IndexSchema.readSchema [solrj_collection_shard2_replica1] Schema name=test
   [junit4]   2> 372247 T1091 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 372270 T1091 oass.IndexSchema.readSchema [solrj_collection_shard1_replica2] Schema name=test
   [junit4]   2> 373068 T1052 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 373094 T1052 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 373099 T1075 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 373099 T1052 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 373113 T1075 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 373120 T1075 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 373132 T1091 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 373135 T1105 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 373138 T1052 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 373145 T1091 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 373148 T1052 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 373148 T1105 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 373150 T1091 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 373154 T1105 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 373155 T1075 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 373157 T1052 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 373160 T1052 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 373163 T1052 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 373164 T1075 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 373164 T1052 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 373167 T1052 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 373168 T1052 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 373169 T1052 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica1' using configuration from collection solrj_collection
   [junit4]   2> 373169 T1052 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 373170 T1052 oasc.SolrCore.<init> [solrj_collection_shard1_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/solrj_collection_shard1_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/solrj_collection_shard1_replica1/data/
   [junit4]   2> 373170 T1052 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17cd4b11
   [junit4]   2> 373172 T1052 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/solrj_collection_shard1_replica1/data
   [junit4]   2> 373173 T1052 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/solrj_collection_shard1_replica1/data/index/
   [junit4]   2> 373174 T1052 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/solrj_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 373173 T1075 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 373175 T1052 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-002/solrj_collection_shard1_replica1/data/index
   [junit4]   2> 373176 T1052 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6c17f63e
   [junit4]   2> 373177 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 373178 T1052 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-5f01aa21))),segFN=segments_1,generation=1}
   [junit4]   2> 373178 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 373178 T1052 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 373179 T1075 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 373181 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 373181 T1075 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 373182 T1075 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica1' using configuration from collection solrj_collection
   [junit4]   2> 373182 T1075 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 373183 T1091 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 373183 T1105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 373183 T1075 oasc.SolrCore.<init> [solrj_collection_shard2_replica1] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/solrj_collection_shard2_replica1/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/solrj_collection_shard2_replica1/data/
   [junit4]   2> 373184 T1075 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17cd4b11
   [junit4]   2> 373186 T1075 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/solrj_collection_shard2_replica1/data
   [junit4]   2> 373186 T1052 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 373187 T1075 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/solrj_collection_shard2_replica1/data/index/
   [junit4]   2> 373187 T1052 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 373188 T1075 oasc.SolrCore.initIndex WARN [solrj_collection_shard2_replica1] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/solrj_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 373188 T1052 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 373189 T1052 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 373190 T1075 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-003/solrj_collection_shard2_replica1/data/index
   [junit4]   2> 373190 T1052 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 373192 T1091 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 373190 T1075 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@59ced1e9
   [junit4]   2> 373192 T1052 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 373194 T1105 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 373194 T1052 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 373195 T1075 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RateLimitedDirectoryWrapper(RAMDirectory(id=lucene-4719f96b))),segFN=segments_1,generation=1}
   [junit4]   2> 373195 T1052 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 373195 T1075 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 373196 T1052 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 373197 T1052 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 373198 T1052 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 373199 T1052 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 373199 T1091 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 373200 T1105 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 373200 T1052 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 373202 T1052 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 373202 T1052 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 373203 T1075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 373203 T1052 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 373203 T1075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 373203 T1091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 373204 T1105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 373204 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 373204 T1052 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 373206 T1075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 373207 T1052 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 373205 T1105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 373208 T1052 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 373205 T1091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 373209 T1052 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 373208 T1105 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 373207 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 373210 T1091 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 373211 T1075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 373213 T1075 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 373213 T1075 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 373214 T1075 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 373214 T1091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 373215 T1105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 373215 T1091 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 373215 T1075 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 373216 T1091 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard1_replica2' using configuration from collection solrj_collection
   [junit4]   2> 373216 T1105 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 373217 T1091 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 373217 T1075 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 373219 T1091 oasc.SolrCore.<init> [solrj_collection_shard1_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/solrj_collection_shard1_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/solrj_collection_shard1_replica2/data/
   [junit4]   2> 373218 T1105 oasc.CoreContainer.create Creating SolrCore 'solrj_collection_shard2_replica2' using configuration from collection solrj_collection
   [junit4]   2> 373220 T1091 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17cd4b11
   [junit4]   2> 373219 T1075 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 373221 T1105 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 373222 T1091 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/solrj_collection_shard1_replica2/data
   [junit4]   2> 373224 T1091 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/solrj_collection_shard1_replica2/data/index/
   [junit4]   2> 373224 T1091 oasc.SolrCore.initIndex WARN [solrj_collection_shard1_replica2] Solr index directory '/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/solrj_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 373222 T1105 oasc.SolrCore.<init> [solrj_collection_shard2_replica2] Opening new SolrCore at /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/solrj_collection_shard2_replica2/, dataDir=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/solrj_collection_shard2_replica2/data/
   [junit4]   2> 373222 T1075 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 373227 T1105 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@17cd4b11
   [junit4]   2> 373226 T1091 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-004/solrj_collection_shard1_replica2/data/index
   [junit4]   2> 373228 T1075 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 373230 T1075 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 373231 T1075 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 373231 T1075 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 373232 T1075 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 373233 T1075 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 373233 T1075 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 373235 T1105 oasc.CachingDirectoryFactory.get return new directory for /usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/solrj_collection_shard2_replica2/data
   [junit4]   2> 373235 T1091 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@a173ddc
   [junit4]   2> 373236 T1105 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build/solr-core/test/J3/temp/solr.cloud.CollectionsAPIDistributedZkTest-8C83B33DBA7C8E1C-001/tempDir-005/solrj_collection_shard2_replica2/data/index/
   [junit4]   2> 373237 T1105 oasc.SolrCore.initIndex WARN [solrj_collection_

[...truncated too long message...]

kRandom), 20parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 2child1_s=PostingsFormat(name=Memory doPackFST= false), 22parent_f2_s=PostingsFormat(name=MockRandom), 2parent_f1_s=OrdsLucene41(blocksize=128), 5grand3_s=PostingsFormat(name=Lucene41WithOrds), rnd_b=PostingsFormat(name=Memory doPackFST= false), _version_=OrdsLucene41(blocksize=128), 43parent_f1_s=PostingsFormat(name=MockRandom), 32parent_f2_s=PostingsFormat(name=Lucene41WithOrds), id=PostingsFormat(name=Memory doPackFST= false), 18parent_f1_s=PostingsFormat(name=MockRandom), 51parent_f2_s=PostingsFormat(name=MockRandom), 6grand3_s=PostingsFormat(name=MockRandom), 8grand6_s=PostingsFormat(name=Lucene41WithOrds), 5parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 9grand0_s=PostingsFormat(name=MockRandom), 6parent_f1_s=OrdsLucene41(blocksize=128), 10parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 47parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 36parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 35parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 28parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 21parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 28parent_f1_s=OrdsLucene41(blocksize=128), 3grand1_s=PostingsFormat(name=Lucene41WithOrds), 8grand0_s=OrdsLucene41(blocksize=128), 25parent_f1_s=PostingsFormat(name=MockRandom), 33parent_f2_s=PostingsFormat(name=MockRandom), 9grand1_s=PostingsFormat(name=Lucene41WithOrds), 50parent_f1_s=PostingsFormat(name=MockRandom), 32parent_f1_s=PostingsFormat(name=MockRandom), 10parent_f1_s=PostingsFormat(name=MockRandom), 29parent_f1_s=PostingsFormat(name=MockRandom), 5parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 8parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 39parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 4grand0_s=OrdsLucene41(blocksize=128), 9parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 3grand2_s=PostingsFormat(name=MockRandom), 26parent_f2_s=PostingsFormat(name=MockRandom), 9parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 5child3_s=OrdsLucene41(blocksize=128), _root_=OrdsLucene41(blocksize=128), 2parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 7grand5_s=PostingsFormat(name=Lucene41WithOrds), 9grand5_s=PostingsFormat(name=Lucene41WithOrds), 26parent_f1_s=OrdsLucene41(blocksize=128), 23parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 37parent_f1_s=OrdsLucene41(blocksize=128), 8grand5_s=PostingsFormat(name=MockRandom), 8parent_f2_s=PostingsFormat(name=MockRandom), 41parent_f2_s=OrdsLucene41(blocksize=128), 7grand0_s=PostingsFormat(name=Memory doPackFST= false), 44parent_f1_s=OrdsLucene41(blocksize=128), 5grand1_s=OrdsLucene41(blocksize=128), 3parent_f2_s=OrdsLucene41(blocksize=128), 21parent_f1_s=PostingsFormat(name=MockRandom), 5grand2_s=PostingsFormat(name=Memory doPackFST= false), 4child0_s=PostingsFormat(name=Lucene41WithOrds), 52parent_f2_s=OrdsLucene41(blocksize=128), 2grand1_s=PostingsFormat(name=Memory doPackFST= false), 51parent_f1_s=OrdsLucene41(blocksize=128), 29parent_f2_s=OrdsLucene41(blocksize=128), 1grand0_s=PostingsFormat(name=MockRandom), 3parent_f1_s=PostingsFormat(name=MockRandom), 17parent_f1_s=OrdsLucene41(blocksize=128), 34parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 47parent_f1_s=PostingsFormat(name=MockRandom), 23parent_f2_s=OrdsLucene41(blocksize=128), 6grand4_s=PostingsFormat(name=Lucene41WithOrds), 4grand3_s=PostingsFormat(name=Memory doPackFST= false), 15parent_f2_s=PostingsFormat(name=MockRandom), 7parent_f2_s=OrdsLucene41(blocksize=128), 41parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 7grand4_s=PostingsFormat(name=Memory doPackFST= false), 48parent_f1_s=OrdsLucene41(blocksize=128), 6grand0_s=PostingsFormat(name=Lucene41WithOrds), 46parent_f2_s=PostingsFormat(name=Memory doPackFST= false), a_t=PostingsFormat(name=Lucene41WithOrds), 4parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 6parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 5grand0_s=PostingsFormat(name=MockRandom), 24parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 18parent_f2_s=OrdsLucene41(blocksize=128), 31parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 34parent_f2_s=OrdsLucene41(blocksize=128), 9grand2_s=PostingsFormat(name=Memory doPackFST= false), 43parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 5child2_s=PostingsFormat(name=Memory doPackFST= false), 7grand6_s=PostingsFormat(name=MockRandom), 24parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 12parent_f2_s=OrdsLucene41(blocksize=128), 7grand2_s=PostingsFormat(name=MockRandom), 27parent_f2_s=OrdsLucene41(blocksize=128), 35parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 6grand5_s=PostingsFormat(name=Memory doPackFST= false), 5child0_s=PostingsFormat(name=MockRandom), 49parent_f2_s=OrdsLucene41(blocksize=128), 38parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 36parent_f1_s=PostingsFormat(name=MockRandom), 5child4_s=PostingsFormat(name=MockRandom), 45parent_f2_s=OrdsLucene41(blocksize=128), 14parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 12parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 30parent_f2_s=OrdsLucene41(blocksize=128), 1child0_s=PostingsFormat(name=MockRandom), 53parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 39parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 7grand3_s=OrdsLucene41(blocksize=128), 42parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 9grand7_s=OrdsLucene41(blocksize=128), 4parent_f2_s=PostingsFormat(name=MockRandom), 2grand0_s=OrdsLucene41(blocksize=128), 3child1_s=PostingsFormat(name=Lucene41WithOrds), 8grand3_s=PostingsFormat(name=Memory doPackFST= false), text_t=OrdsLucene41(blocksize=128), 8grand4_s=OrdsLucene41(blocksize=128), 19parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 4child2_s=OrdsLucene41(blocksize=128), 37parent_f2_s=PostingsFormat(name=MockRandom), 9grand4_s=PostingsFormat(name=MockRandom), 30parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 5grand4_s=PostingsFormat(name=MockRandom), 38parent_f2_s=OrdsLucene41(blocksize=128), 13parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 7parent_f1_s=PostingsFormat(name=MockRandom), 15parent_f1_s=OrdsLucene41(blocksize=128), 50parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 7grand1_s=PostingsFormat(name=Lucene41WithOrds), 25parent_f2_s=PostingsFormat(name=Lucene41WithOrds), 40parent_f1_s=OrdsLucene41(blocksize=128), 16parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 45parent_f1_s=PostingsFormat(name=Memory doPackFST= false), 19parent_f2_s=PostingsFormat(name=MockRandom), 0parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 20parent_f2_s=PostingsFormat(name=Memory doPackFST= false), 31parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 4child1_s=PostingsFormat(name=MockRandom), 40parent_f2_s=PostingsFormat(name=MockRandom), 6grand1_s=PostingsFormat(name=Memory doPackFST= false), 46parent_f1_s=PostingsFormat(name=Lucene41WithOrds), 0parent_f2_s=PostingsFormat(name=MockRandom), 3child0_s=PostingsFormat(name=Memory doPackFST= false), 3grand0_s=PostingsFormat(name=Memory doPackFST= false), type_s=OrdsLucene41(blocksize=128), 6grand2_s=OrdsLucene41(blocksize=128), 8grand1_s=PostingsFormat(name=MockRandom), 9grand3_s=OrdsLucene41(blocksize=128), 14parent_f1_s=PostingsFormat(name=MockRandom)}, docValues:{}, sim=DefaultSimilarity, locale=sv, timezone=Atlantic/Jan_Mayen
   [junit4]   2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=16,threads=1,free=171661496,total=483917824
   [junit4]   2> NOTE: All tests run in this JVM: [ShowFileRequestHandlerTest, TestFieldCollectionResource, ResponseHeaderTest, TestHighFrequencyDictionaryFactory, TestZkChroot, TestMissingGroups, DistributedDebugComponentTest, SolrIndexConfigTest, ZkControllerTest, TestConfig, OpenCloseCoreStressTest, DirectSolrSpellCheckerTest, TestFieldSortValues, TestStressVersions, ClusterStateTest, TestMaxScoreQueryParser, TestFastOutputStream, VMParamsZkACLAndCredentialsProvidersTest, TestRequestStatusCollectionAPI, ExitableDirectoryReaderTest, AnalyticsMergeStrategyTest, ExternalFileFieldSortTest, PrimitiveFieldTypeTest, TestDefaultSearchFieldResource, CurrencyFieldOpenExchangeTest, RequiredFieldsTest, TestRecovery, TestSolrXmlPersistence, HdfsUnloadDistributedZkTest, FullSolrCloudDistribCmdsTest]
   [junit4] Completed on J1 in 345.34s, 1 test, 1 error <<< FAILURES!

[...truncated 946 lines...]
BUILD FAILED
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:531: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:472: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/build.xml:61: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/extra-targets.xml:39: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/solr/common-build.xml:496: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:1359: The following error occurred while executing this line:
/usr/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-trunk/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 2 errors, 54 ignored (10 assumptions)

Total time: 219 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Sending artifact delta relative to Lucene-Solr-NightlyTests-trunk #636
Archived 5 artifacts
Archive block size is 32768
Received 0 blocks and 529924994 bytes
Compression is 0.0%
Took 1 min 32 sec
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure