You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/09/29 16:33:54 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_67) - Build # 11349 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11349/
Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch

Error Message:
no exception matching expected: 400: Request took too long during query expansion. Terminating request.

Stack Trace:
java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request.
	at __randomizedtesting.SeedInfo.seed([19D3E66939E34CC4:983568714EBC2CF8]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:75)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11734 lines...]
   [junit4] Suite: org.apache.solr.cloud.CloudExitableDirectoryReaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/init-core-data-001
   [junit4]   2> 2312229 T6598 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2312229 T6598 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2312232 T6598 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2312232 T6598 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2312233 T6599 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2312333 T6598 oasc.ZkTestServer.run start zk server on port:41961
   [junit4]   2> 2312333 T6598 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2312334 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312336 T6605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78a50424 name:ZooKeeperConnection Watcher:127.0.0.1:41961 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312336 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312337 T6598 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2312337 T6598 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2312340 T6598 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2312341 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312342 T6607 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@43dcfed0 name:ZooKeeperConnection Watcher:127.0.0.1:41961/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312342 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312342 T6598 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2312342 T6598 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2312344 T6598 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2312346 T6598 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2312347 T6598 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2312348 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2312348 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2312350 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2312350 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2312351 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2312352 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2312353 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2312353 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2312354 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2312354 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2312355 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2312356 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2312357 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2312357 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2312358 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2312358 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2312359 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2312359 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2312360 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2312361 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2312362 T6598 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2312362 T6598 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2312584 T6598 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2312587 T6598 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43031
   [junit4]   2> 2312588 T6598 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2312588 T6598 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2312588 T6598 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002
   [junit4]   2> 2312589 T6598 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002/'
   [junit4]   2> 2312600 T6598 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002/solr.xml
   [junit4]   2> 2312633 T6598 oasc.CoreContainer.<init> New CoreContainer 1374006271
   [junit4]   2> 2312633 T6598 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002/]
   [junit4]   2> 2312634 T6598 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2312634 T6598 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2312634 T6598 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2312634 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2312635 T6598 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2312635 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2312635 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2312635 T6598 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2312635 T6598 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2312636 T6598 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2312636 T6598 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2312636 T6598 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2312636 T6598 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2312637 T6598 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41961/solr
   [junit4]   2> 2312637 T6598 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2312637 T6598 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2312638 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312639 T6618 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d7f1c5a name:ZooKeeperConnection Watcher:127.0.0.1:41961 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312639 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312639 T6598 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2312641 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2312642 T6620 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e5ebc8e name:ZooKeeperConnection Watcher:127.0.0.1:41961/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2312642 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2312644 T6598 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2312648 T6598 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2312650 T6598 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2312652 T6598 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2312653 T6598 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2312654 T6598 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2312655 T6598 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43031_
   [junit4]   2> 2312656 T6598 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43031_
   [junit4]   2> 2312657 T6598 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2312658 T6598 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2312658 T6598 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2312659 T6598 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:43031_
   [junit4]   2> 2312660 T6598 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2312660 T6598 oasc.Overseer.start Overseer (id=92536876425674755-127.0.0.1:43031_-n_0000000000) starting
   [junit4]   2> 2312662 T6598 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2312666 T6598 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2312666 T6598 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2312666 T6622 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2312668 T6598 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2312670 T6598 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2312672 T6621 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2312675 T6624 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2312675 T6624 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2312676 T6624 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2312676 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2312679 T6621 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43031",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:43031_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection"}
   [junit4]   2> 2312680 T6621 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2312680 T6621 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 2312680 T6621 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2312683 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2313677 T6624 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2313677 T6624 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2313678 T6624 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2313678 T6624 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2313679 T6624 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2313680 T6624 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002/collection1/'
   [junit4]   2> 2313680 T6624 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2313681 T6624 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2313681 T6624 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2313718 T6624 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2313743 T6624 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2313746 T6624 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2313752 T6624 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2313987 T6624 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2313991 T6624 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2313992 T6624 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2313999 T6624 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2314001 T6624 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2314003 T6624 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2314004 T6624 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2314004 T6624 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2314004 T6624 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2314005 T6624 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2314005 T6624 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2314005 T6624 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2314005 T6624 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2314005 T6624 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/control/data/
   [junit4]   2> 2314005 T6624 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16e0f6ab
   [junit4]   2> 2314006 T6624 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/control/data
   [junit4]   2> 2314006 T6624 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/control/data/index/
   [junit4]   2> 2314006 T6624 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2314006 T6624 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/control/data/index
   [junit4]   2> 2314007 T6624 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2314007 T6624 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-2e5bd3e)),segFN=segments_1,generation=1}
   [junit4]   2> 2314007 T6624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2314010 T6624 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2314010 T6624 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2314010 T6624 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2314010 T6624 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2314010 T6624 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2314011 T6624 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2314011 T6624 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2314011 T6624 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2314011 T6624 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2314012 T6624 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2314012 T6624 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2314012 T6624 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2314013 T6624 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2314013 T6624 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2314013 T6624 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2314013 T6624 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2314014 T6624 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2314014 T6624 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2314014 T6624 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2314015 T6624 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2314017 T6624 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2314018 T6624 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2314019 T6624 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2314020 T6624 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2314022 T6624 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2314022 T6624 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2314023 T6624 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=19.7548828125, floorSegmentMB=1.4580078125, forceMergeDeletesPctAllowed=17.616635709406662, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20298112688796532
   [junit4]   2> 2314023 T6624 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-2e5bd3e)),segFN=segments_1,generation=1}
   [junit4]   2> 2314023 T6624 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2314024 T6624 oass.SolrIndexSearcher.<init> Opening Searcher@65138a8b[collection1] main
   [junit4]   2> 2314024 T6624 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2314025 T6624 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2314025 T6624 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2314026 T6624 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2314026 T6624 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2314026 T6624 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2314027 T6624 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2314027 T6624 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2314027 T6624 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2314029 T6624 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2314029 T6624 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2314029 T6624 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2314031 T6626 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@65138a8b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2314032 T6624 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2314033 T6629 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43031 collection:control_collection shard:shard1
   [junit4]   2> 2314033 T6598 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2314034 T6598 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2314034 T6629 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2314034 T6598 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2314035 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2314036 T6631 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@49c9c229 name:ZooKeeperConnection Watcher:127.0.0.1:41961/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2314036 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2314037 T6598 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2314037 T6598 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2314038 T6629 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2314039 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2314040 T6598 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2314040 T6629 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2314040 T6629 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3013 name=collection1 org.apache.solr.core.SolrCore@2b291673 url=http://127.0.0.1:43031/collection1 node=127.0.0.1:43031_ C3013_STATE=coll:control_collection core:collection1 props:{base_url=http://127.0.0.1:43031, core=collection1, state=down, node_name=127.0.0.1:43031_}
   [junit4]   2> 2314040 T6629 C3013 P43031 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43031/collection1/
   [junit4]   2> 2314040 T6629 C3013 P43031 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2314041 T6629 C3013 P43031 oasc.SyncStrategy.syncToMe http://127.0.0.1:43031/collection1/ has no replicas
   [junit4]   2> 2314041 T6629 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43031/collection1/ shard1
   [junit4]   2> 2314041 T6629 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2314044 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2314147 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2314147 T6632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2314195 T6629 oasc.ZkController.register We are http://127.0.0.1:43031/collection1/ and leader is http://127.0.0.1:43031/collection1/
   [junit4]   2> 2314195 T6629 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43031
   [junit4]   2> 2314195 T6629 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2314195 T6629 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2314195 T6629 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2314197 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2314199 T6621 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:43031",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1:43031_",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"}
   [junit4]   2> 2314262 T6598 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2314262 T6598 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2314264 T6598 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39033
   [junit4]   2> 2314265 T6598 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2314265 T6598 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2314265 T6598 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003
   [junit4]   2> 2314265 T6598 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003/'
   [junit4]   2> 2314276 T6598 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003/solr.xml
   [junit4]   2> 2314305 T6632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2314305 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2314315 T6598 oasc.CoreContainer.<init> New CoreContainer 16828809
   [junit4]   2> 2314316 T6598 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003/]
   [junit4]   2> 2314316 T6598 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2314317 T6598 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2314317 T6598 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2314317 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2314317 T6598 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2314318 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2314318 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2314318 T6598 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2314318 T6598 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2314319 T6598 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2314319 T6598 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2314319 T6598 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2314320 T6598 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2314320 T6598 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41961/solr
   [junit4]   2> 2314320 T6598 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2314321 T6598 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2314321 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2314322 T6643 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aaf41e2 name:ZooKeeperConnection Watcher:127.0.0.1:41961 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2314323 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2314323 T6598 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2314325 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2314327 T6645 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3bee5520 name:ZooKeeperConnection Watcher:127.0.0.1:41961/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2314327 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2314334 T6598 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2315337 T6598 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39033_
   [junit4]   2> 2315339 T6598 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39033_
   [junit4]   2> 2315352 T6598 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2315358 T6647 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2315359 T6647 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2315360 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2315360 T6647 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2315362 T6621 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39033",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:39033_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 2315363 T6621 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2315363 T6621 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
   [junit4]   2> 2315363 T6621 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2315466 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2315466 T6646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2315466 T6632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2316360 T6647 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2316361 T6647 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2316361 T6647 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2316362 T6647 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2316363 T6647 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2316363 T6647 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003/collection1/'
   [junit4]   2> 2316364 T6647 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2316364 T6647 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2316364 T6647 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2316396 T6647 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2316416 T6647 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2316417 T6647 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2316422 T6647 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2316638 T6647 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2316642 T6647 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2316644 T6647 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2316652 T6647 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2316655 T6647 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2316657 T6647 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2316657 T6647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2316658 T6647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2316658 T6647 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2316658 T6647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2316658 T6647 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2316659 T6647 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2316659 T6647 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2316659 T6647 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty1/
   [junit4]   2> 2316659 T6647 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16e0f6ab
   [junit4]   2> 2316660 T6647 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty1
   [junit4]   2> 2316660 T6647 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty1/index/
   [junit4]   2> 2316660 T6647 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2316661 T6647 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty1/index
   [junit4]   2> 2316661 T6647 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2316662 T6647 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-7538d3e2)),segFN=segments_1,generation=1}
   [junit4]   2> 2316662 T6647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2316665 T6647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2316666 T6647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2316666 T6647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2316666 T6647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2316666 T6647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2316666 T6647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2316667 T6647 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2316667 T6647 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2316667 T6647 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2316668 T6647 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2316668 T6647 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2316668 T6647 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2316669 T6647 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2316669 T6647 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2316669 T6647 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2316670 T6647 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2316670 T6647 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2316670 T6647 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2316671 T6647 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2316671 T6647 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2316674 T6647 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2316675 T6647 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2316676 T6647 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2316677 T6647 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2316679 T6647 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2316680 T6647 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2316680 T6647 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=19.7548828125, floorSegmentMB=1.4580078125, forceMergeDeletesPctAllowed=17.616635709406662, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20298112688796532
   [junit4]   2> 2316681 T6647 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-7538d3e2)),segFN=segments_1,generation=1}
   [junit4]   2> 2316681 T6647 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2316681 T6647 oass.SolrIndexSearcher.<init> Opening Searcher@28da9344[collection1] main
   [junit4]   2> 2316681 T6647 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2316682 T6647 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2316682 T6647 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2316683 T6647 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2316683 T6647 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2316683 T6647 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2316684 T6647 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2316684 T6647 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2316684 T6647 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2316684 T6647 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2316688 T6648 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@28da9344[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2316691 T6647 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2316692 T6651 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39033 collection:collection1 shard:shard2
   [junit4]   2> 2316692 T6598 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2316693 T6598 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2316693 T6651 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2316698 T6651 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2316700 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2316700 T6651 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2316700 T6651 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3014 name=collection1 org.apache.solr.core.SolrCore@58e2e69f url=http://127.0.0.1:39033/collection1 node=127.0.0.1:39033_ C3014_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:39033, core=collection1, state=down, node_name=127.0.0.1:39033_}
   [junit4]   2> 2316700 T6651 C3014 P39033 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39033/collection1/
   [junit4]   2> 2316701 T6651 C3014 P39033 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2316701 T6651 C3014 P39033 oasc.SyncStrategy.syncToMe http://127.0.0.1:39033/collection1/ has no replicas
   [junit4]   2> 2316701 T6651 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39033/collection1/ shard2
   [junit4]   2> 2316701 T6651 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2316704 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2316806 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2316806 T6632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2316806 T6646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2316855 T6651 oasc.ZkController.register We are http://127.0.0.1:39033/collection1/ and leader is http://127.0.0.1:39033/collection1/
   [junit4]   2> 2316855 T6651 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39033
   [junit4]   2> 2316855 T6651 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2316855 T6651 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2316855 T6651 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2316856 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2316858 T6621 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39033",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1:39033_",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 2316918 T6598 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2316919 T6598 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2316921 T6598 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54360
   [junit4]   2> 2316923 T6598 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2316923 T6598 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2316923 T6598 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004
   [junit4]   2> 2316923 T6598 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004/'
   [junit4]   2> 2316933 T6598 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004/solr.xml
   [junit4]   2> 2316960 T6646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2316960 T6632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2316960 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2316974 T6598 oasc.CoreContainer.<init> New CoreContainer 203455862
   [junit4]   2> 2316974 T6598 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004/]
   [junit4]   2> 2316975 T6598 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2316975 T6598 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2316976 T6598 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2316976 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2316976 T6598 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2316977 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2316977 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2316977 T6598 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2316978 T6598 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2316978 T6598 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2316979 T6598 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2316979 T6598 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2316980 T6598 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2316980 T6598 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41961/solr
   [junit4]   2> 2316980 T6598 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2316981 T6598 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2316982 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2316984 T6662 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@290cd432 name:ZooKeeperConnection Watcher:127.0.0.1:41961 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2316984 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2316984 T6598 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2316987 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2316988 T6664 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66c85e55 name:ZooKeeperConnection Watcher:127.0.0.1:41961/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2316988 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2316995 T6598 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2317998 T6598 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54360_
   [junit4]   2> 2317999 T6598 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54360_
   [junit4]   2> 2318003 T6598 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2318009 T6666 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2318009 T6666 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2318010 T6666 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2318010 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2318014 T6621 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54360",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "node_name":"127.0.0.1:54360_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 2318015 T6621 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 2318015 T6621 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2318118 T6632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2318118 T6646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2318118 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2318118 T6665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319011 T6666 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2319011 T6666 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2319012 T6666 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2319012 T6666 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2319013 T6666 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2319014 T6666 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004/collection1/'
   [junit4]   2> 2319015 T6666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2319015 T6666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2319016 T6666 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2319044 T6666 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2319085 T6666 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2319087 T6666 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2319095 T6666 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2319278 T6666 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2319282 T6666 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2319283 T6666 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2319292 T6666 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2319294 T6666 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2319297 T6666 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2319298 T6666 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2319298 T6666 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2319298 T6666 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2319299 T6666 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2319299 T6666 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2319299 T6666 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2319299 T6666 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2319300 T6666 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty2/
   [junit4]   2> 2319300 T6666 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@16e0f6ab
   [junit4]   2> 2319301 T6666 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty2
   [junit4]   2> 2319301 T6666 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty2/index/
   [junit4]   2> 2319301 T6666 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2319301 T6666 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty2/index
   [junit4]   2> 2319302 T6666 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2319302 T6666 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-51b00c42)),segFN=segments_1,generation=1}
   [junit4]   2> 2319302 T6666 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2319305 T6666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2319305 T6666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2319306 T6666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2319306 T6666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2319306 T6666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2319306 T6666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2319307 T6666 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2319307 T6666 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2319307 T6666 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2319307 T6666 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2319308 T6666 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2319308 T6666 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2319309 T6666 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2319309 T6666 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2319309 T6666 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2319309 T6666 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2319310 T6666 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2319310 T6666 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2319310 T6666 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2319311 T6666 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2319313 T6666 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2319314 T6666 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2319315 T6666 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2319316 T6666 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2319317 T6666 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2319317 T6666 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2319318 T6666 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=19.7548828125, floorSegmentMB=1.4580078125, forceMergeDeletesPctAllowed=17.616635709406662, segmentsPerTier=36.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20298112688796532
   [junit4]   2> 2319318 T6666 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-51b00c42)),segFN=segments_1,generation=1}
   [junit4]   2> 2319318 T6666 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2319319 T6666 oass.SolrIndexSearcher.<init> Opening Searcher@27a7c7ae[collection1] main
   [junit4]   2> 2319319 T6666 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2319320 T6666 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2319320 T6666 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2319321 T6666 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2319321 T6666 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2319321 T6666 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2319321 T6666 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2319322 T6666 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2319322 T6666 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2319322 T6666 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2319324 T6667 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@27a7c7ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2319326 T6666 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2319327 T6670 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54360 collection:collection1 shard:shard1
   [junit4]   2> 2319327 T6598 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2319327 T6598 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2319328 T6670 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2319331 T6670 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2319331 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2319332 T6670 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2319332 T6670 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3015 name=collection1 org.apache.solr.core.SolrCore@502d69ba url=http://127.0.0.1:54360/collection1 node=127.0.0.1:54360_ C3015_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:54360, core=collection1, state=down, node_name=127.0.0.1:54360_}
   [junit4]   2> 2319332 T6670 C3015 P54360 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54360/collection1/
   [junit4]   2> 2319332 T6670 C3015 P54360 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2319332 T6670 C3015 P54360 oasc.SyncStrategy.syncToMe http://127.0.0.1:54360/collection1/ has no replicas
   [junit4]   2> 2319332 T6670 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54360/collection1/ shard1
   [junit4]   2> 2319333 T6670 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2319335 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2319437 T6665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319437 T6632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319437 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319437 T6646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319486 T6670 oasc.ZkController.register We are http://127.0.0.1:54360/collection1/ and leader is http://127.0.0.1:54360/collection1/
   [junit4]   2> 2319486 T6670 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54360
   [junit4]   2> 2319486 T6670 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2319486 T6670 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2319486 T6670 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2319488 T6625 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2319490 T6621 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54360",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "node_name":"127.0.0.1:54360_",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"}
   [junit4]   2> 2319572 T6598 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2319573 T6598 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2319575 T6598 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55891
   [junit4]   2> 2319576 T6598 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2319576 T6598 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2319576 T6598 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-005
   [junit4]   2> 2319577 T6598 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-005/'
   [junit4]   2> 2319587 T6598 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-005/solr.xml
   [junit4]   2> 2319607 T6646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319607 T6625 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319607 T6665 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319607 T6632 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2319609 T6598 oasc.CoreContainer.<init> New CoreContainer 1755637414
   [junit4]   2> 2319609 T6598 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-005/]
   [junit4]   2> 2319610 T6598 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2319610 T6598 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2319610 T6598 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2319611 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2319611 T6598 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2319611 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2319611 T6598 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2319611 T6598 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2319612 T6598 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2319612 T6598 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2319612 T6598 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2319612 T6598 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2319613 T6598 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2319613 T6598 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41961/solr
   [junit4]   2> 2319613 T6598 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2319613 T6598 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2319614 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2319616 T6681 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c067efc name:ZooKeeperConnection Watcher:127.0.0.1:41961 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2319616 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2319616 T6598 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2319618 T6598 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2319620 T6683 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b5b5d28 name:ZooKeeperConnection Watcher:127.0.0.1:41961/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2319620 T6598 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2319626 T6598 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2320629 T6598 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55891_
   [junit4]   2> 2320630 T6598 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55891_
   [junit4]   2> 2320635 T6598 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2320643 T6685 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2320643 T6685 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2320644 T6685 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2320644 T6625 oasc.DistributedQueue$LatchChildWatcher.p

[...truncated too long message...]

tory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001/tempDir-001/jetty4/index
   [junit4]   2> 2396527 T6598 oasc.Overseer.close Overseer (id=92536876425674764-127.0.0.1:48545_-n_0000000004) closing
   [junit4]   2> 2396527 T6742 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:48545_
   [junit4]   2> 2398032 T6737 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 2398057 T6598 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 2403046 T6598 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 2403048 T6598 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:41961 41961
   [junit4]   2> 2403468 T6599 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:41961 41961
   [junit4]   2> 2403470 T6599 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/aliases.json
   [junit4]   2> 		6	/solr/clusterstate.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 		4	/solr/overseer/queue
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CloudExitableDirectoryReaderTest -Dtests.method=testDistribSearch -Dtests.seed=19D3E66939E34CC4 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_IE -Dtests.timezone=ART -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 91.3s J1 | CloudExitableDirectoryReaderTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([19D3E66939E34CC4:983568714EBC2CF8]:0)
   [junit4]    > 	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
   [junit4]    > 	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:75)
   [junit4]    > 	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2403490 T6598 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.CloudExitableDirectoryReaderTest-19D3E66939E34CC4-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=en_IE, timezone=ART
   [junit4]   2> NOTE: Linux 3.13.0-36-generic amd64/Oracle Corporation 1.7.0_67 (64-bit)/cpus=8,threads=1,free=54871448,total=222527488
   [junit4]   2> NOTE: All tests run in this JVM: [OutputWriterTest, ScriptEngineTest, TestPHPSerializedResponseWriter, SchemaVersionSpecificBehaviorTest, TestNoOpRegenerator, TestJoin, DistributedQueryComponentCustomSortTest, SuggestComponentTest, CursorMarkTest, HdfsSyncSliceTest, DeleteInactiveReplicaTest, StressHdfsTest, TestInfoStreamLogging, SolrCoreTest, TestNRTOpen, CoreContainerCoreInitFailuresTest, ResponseHeaderTest, TestPseudoReturnFields, TestRecovery, HdfsDirectoryTest, TestManagedResourceStorage, TestCopyFieldCollectionResource, AnalyticsQueryTest, TestFiltering, TestIBSimilarityFactory, DistribDocExpirationUpdateProcessorTest, TestCollationField, BadCopyFieldTest, SpatialFilterTest, ChaosMonkeySafeLeaderTest, SharedFSAutoReplicaFailoverTest, TestDistributedGrouping, TestAtomicUpdateErrorCases, ModifyConfFileTest, TestRealTimeGet, LoggingHandlerTest, ShardRoutingCustomTest, StatsComponentTest, TestLazyCores, TestCSVLoader, AlternateDirectoryTest, PreAnalyzedFieldTest, FullSolrCloudDistribCmdsTest, TestRestManager, CopyFieldTest, SimplePostToolTest, UniqFieldsUpdateProcessorFactoryTest, ZkSolrClientTest, TestFieldSortValues, SolrIndexConfigTest, TestSolrXmlPersistence, ChangedSchemaMergeTest, CoreMergeIndexesAdminHandlerTest, OpenCloseCoreStressTest, RollingRestartTest, TestPhraseSuggestions, DistributedQueryComponentOptimizationTest, TestReRankQParserPlugin, SoftAutoCommitTest, TestMultiCoreConfBootstrap, AssignTest, TestFieldTypeCollectionResource, MigrateRouteKeyTest, AnalyticsMergeStrategyTest, EnumFieldTest, TestComplexPhraseQParserPlugin, TestOmitPositions, DistanceFunctionTest, CurrencyFieldXmlFileTest, UUIDFieldTest, TermVectorComponentTest, TestJmxMonitoredMap, DateMathParserTest, TestFieldCollectionResource, SOLR749Test, UpdateRequestProcessorFactoryTest, TestQueryUtils, TestBinaryField, TestZkChroot, TestTolerantSearch, SearchHandlerTest, TestElisionMultitermQuery, SuggesterFSTTest, TestFieldResource, SolrXmlInZkTest, DefaultValueUpdateProcessorTest, TestNonNRTOpen, TestExtendedDismaxParser, HardAutoCommitTest, TestManagedSchemaDynamicFieldResource, DirectUpdateHandlerOptimizeTest, TestHighlightDedupGrouping, LukeRequestHandlerTest, LeaderElectionIntegrationTest, TestInitParams, MultiTermTest, ResourceLoaderTest, TestSolrQueryParser, TestSchemaNameResource, TestCharFilters, FieldAnalysisRequestHandlerTest, HdfsWriteToMultipleCollectionsTest, SolrRequestParserTest, TestMergePolicyConfig, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, BasicZkTest, RecoveryZkTest, TestRandomFaceting, ZkCLITest, TestDistributedSearch, AutoCommitTest, TestHashPartitioner, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestStressReorder, DistributedTermsComponentTest, TestRangeQuery, TestCoreContainer, SimpleFacetsTest, TestGroupingSearch, QueryElevationComponentTest, ConvertedLegacyTest, TestSort, TestFunctionQuery, BasicFunctionalityTest, TestBadConfig, TestIndexSearcher, HighlighterTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, CoreAdminHandlerTest, DocValuesTest, SuggesterTSTTest, TestStressLucene, PolyFieldTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, TestUpdate, FieldMutatingUpdateProcessorTest, QueryEqualityTest, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, TestRemoteStreaming, DocValuesMultiTest, TestSolrDeletionPolicy1, SolrInfoMBeanTest, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, DisMaxRequestHandlerTest, TestWriterPerf, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, DocumentBuilderTest, TestValueSourceCache, TestIndexingPerformance, RequiredFieldsTest, TestArbitraryIndexDir, FastVectorHighlighterTest, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, SolrPluginUtilsTest, IndexSchemaTest, ReturnFieldsTest, JSONWriterTest, TestCSVResponseWriter, MBeansHandlerTest, QueryParsingTest, JsonLoaderTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, CSVRequestHandlerTest, TestComponentsName, TestBinaryResponseWriter, TestLFUCache, HighlighterConfigTest, UpdateParamsTest, TestSolrIndexConfig, TestQuerySenderNoQuery, TestSolrDeletionPolicy2, SampleTest, TestDocSet, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, TestConfig, TestPostingsSolrHighlighter, DirectSolrConnectionTest, SpellPossibilityIteratorTest, SynonymTokenizerTest, TestXIncludeConfig, TestSweetSpotSimilarityFactory, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TimeZoneUtilsTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, TestSystemIdResolver, PrimUtilsTest, DateFieldTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, TestSolrJ, TestUtils, TestDocumentBuilder, SliceStateTest, FileUtilsTest, CircularListTest, TestRTGBase, DistributedIntervalFacetingTest, TestDistributedMissingSort, AsyncMigrateRouteKeyTest, CloudExitableDirectoryReaderTest]
   [junit4] Completed on J1 in 91.28s, 1 test, 1 failure <<< FAILURES!

[...truncated 297 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:524: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:472: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1359: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 1 failure, 44 ignored (22 assumptions)

Total time: 96 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_67 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b28) - Build # 11350 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11350/
Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseParallelGC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:47512/vqqpo, http://127.0.0.1:59264/vqqpo, http://127.0.0.1:59549/vqqpo, http://127.0.0.1:46014/vqqpo, http://127.0.0.1:56419/vqqpo]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:47512/vqqpo, http://127.0.0.1:59264/vqqpo, http://127.0.0.1:59549/vqqpo, http://127.0.0.1:46014/vqqpo, http://127.0.0.1:56419/vqqpo]
	at __randomizedtesting.SeedInfo.seed([48532332BF07798A:C9B5AD2AC85819B6]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:322)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.sendRequest(CloudSolrServer.java:880)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.requestWithRetryOnStaleState(CloudSolrServer.java:658)
	at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:601)
	at org.apache.solr.cloud.DeleteReplicaTest.removeAndWaitForReplicaGone(DeleteReplicaTest.java:171)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:144)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:88)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:484)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Could not  remove replica : delLiveColl/shard1/core_node1
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:570)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:215)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:211)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.doRequest(LBHttpSolrServer.java:343)
	at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:304)
	... 46 more


REGRESSION:  org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery

Error Message:


Stack Trace:
java.lang.AssertionError: 
	at __randomizedtesting.SeedInfo.seed([48532332BF07798A:FB041770DBDA9D0F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.assertQEx(SolrTestCaseJ4.java:850)
	at org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery(ExitableDirectoryReaderTest.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:484)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch

Error Message:
no exception matching expected: 400: Request took too long during query expansion. Terminating request.

Stack Trace:
java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request.
	at __randomizedtesting.SeedInfo.seed([48532332BF07798A:C9B5AD2AC85819B6]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:484)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11329 lines...]
   [junit4] Suite: org.apache.solr.cloud.CloudExitableDirectoryReaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/init-core-data-001
   [junit4]   2> 1033140 T2886 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 1033140 T2886 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1033143 T2886 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1033144 T2886 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1033145 T2887 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1033244 T2886 oasc.ZkTestServer.run start zk server on port:49267
   [junit4]   2> 1033245 T2886 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1033246 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1033249 T2893 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f807faa name:ZooKeeperConnection Watcher:127.0.0.1:49267 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1033249 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1033250 T2886 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1033250 T2886 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1033253 T2886 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1033258 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1033259 T2895 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d00f7b1 name:ZooKeeperConnection Watcher:127.0.0.1:49267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1033259 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1033260 T2886 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1033260 T2886 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1033262 T2886 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1033264 T2886 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1033264 T2886 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1033266 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1033266 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1033268 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1033268 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1033270 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1033270 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1033272 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1033272 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1033273 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1033274 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1033283 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1033284 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1033285 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1033285 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1033286 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1033287 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1033288 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1033288 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1033289 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1033289 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1033290 T2886 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1033291 T2886 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1033292 T2886 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1033295 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1033302 T2897 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@202ebc20 name:ZooKeeperConnection Watcher:127.0.0.1:49267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1033303 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1033303 T2886 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1033575 T2886 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1033577 T2886 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1033592 T2886 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:34510
   [junit4]   2> 1033593 T2886 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1033593 T2886 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1033594 T2886 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002
   [junit4]   2> 1033594 T2886 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002/'
   [junit4]   2> 1033604 T2886 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002/solr.xml
   [junit4]   2> 1033613 T2886 oasc.CoreContainer.<init> New CoreContainer 1641391552
   [junit4]   2> 1033614 T2886 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002/]
   [junit4]   2> 1033614 T2886 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1033614 T2886 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1033615 T2886 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1033615 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1033615 T2886 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1033615 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1033615 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1033615 T2886 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1033616 T2886 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1033617 T2886 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1033617 T2886 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1033618 T2886 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1033618 T2886 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1033618 T2886 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49267/solr
   [junit4]   2> 1033618 T2886 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1033619 T2886 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1033619 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1033624 T2908 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@70a91049 name:ZooKeeperConnection Watcher:127.0.0.1:49267 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1033624 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1033625 T2886 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1033631 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1033632 T2910 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54b1e127 name:ZooKeeperConnection Watcher:127.0.0.1:49267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1033632 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1033634 T2886 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1033635 T2886 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1033637 T2886 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1033638 T2886 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1033639 T2886 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1033641 T2886 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1033642 T2886 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34510_
   [junit4]   2> 1033642 T2886 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:34510_
   [junit4]   2> 1033644 T2886 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1033645 T2886 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1033646 T2886 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1033647 T2886 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:34510_
   [junit4]   2> 1033647 T2886 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1033648 T2886 oasc.Overseer.start Overseer (id=92537289700737028-127.0.0.1:34510_-n_0000000000) starting
   [junit4]   2> 1033649 T2886 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1033653 T2886 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1033654 T2912 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1033655 T2911 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1033655 T2886 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1033657 T2886 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1033658 T2886 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1033661 T2914 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1033663 T2914 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1033665 T2914 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1033665 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1033667 T2911 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34510",
   [junit4]   2> 	  "node_name":"127.0.0.1:34510_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1033667 T2911 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 1033667 T2911 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 1033667 T2911 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1033683 T2915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1034665 T2914 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1034666 T2914 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1034666 T2914 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1034666 T2914 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1034666 T2914 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1034667 T2914 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002/collection1/'
   [junit4]   2> 1034667 T2914 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1034667 T2914 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1034667 T2914 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1034683 T2914 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1034706 T2914 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1034707 T2914 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1034710 T2914 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1034779 T2914 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1034783 T2914 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1034784 T2914 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1034789 T2914 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1034791 T2914 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1034792 T2914 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1034793 T2914 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1034793 T2914 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1034794 T2914 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1034794 T2914 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1034794 T2914 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1034794 T2914 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1034795 T2914 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1034795 T2914 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/control/data/
   [junit4]   2> 1034795 T2914 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f48baf
   [junit4]   2> 1034795 T2914 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/control/data
   [junit4]   2> 1034795 T2914 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/control/data/index/
   [junit4]   2> 1034795 T2914 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1034796 T2914 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/control/data/index
   [junit4]   2> 1034796 T2914 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1034796 T2914 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-4ba416f7)),segFN=segments_1,generation=1}
   [junit4]   2> 1034796 T2914 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1034798 T2914 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1034798 T2914 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1034798 T2914 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1034798 T2914 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1034798 T2914 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1034799 T2914 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1034799 T2914 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1034799 T2914 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1034799 T2914 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1034800 T2914 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1034800 T2914 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1034800 T2914 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1034800 T2914 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1034800 T2914 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1034800 T2914 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1034801 T2914 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1034801 T2914 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1034801 T2914 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1034801 T2914 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1034801 T2914 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1034803 T2914 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1034803 T2914 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1034804 T2914 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1034804 T2914 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1034805 T2914 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1034805 T2914 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1034806 T2914 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=82.064453125, floorSegmentMB=2.1884765625, forceMergeDeletesPctAllowed=17.518579599039594, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.34514987743752135
   [junit4]   2> 1034806 T2914 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-4ba416f7)),segFN=segments_1,generation=1}
   [junit4]   2> 1034806 T2914 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1034806 T2914 oass.SolrIndexSearcher.<init> Opening Searcher@709ab28d[collection1] main
   [junit4]   2> 1034806 T2914 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1034807 T2914 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1034807 T2914 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1034807 T2914 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1034807 T2914 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1034808 T2914 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1034808 T2914 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1034808 T2914 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1034808 T2914 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 1034809 T2914 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1034810 T2914 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1034810 T2914 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1034811 T2916 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@709ab28d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1034811 T2914 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1034812 T2919 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:34510 collection:control_collection shard:shard1
   [junit4]   2> 1034812 T2886 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1034812 T2919 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1034812 T2886 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1034814 T2886 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1034815 T2919 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1034815 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1034816 T2921 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4146c5a6 name:ZooKeeperConnection Watcher:127.0.0.1:49267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1034817 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1034817 T2886 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1034818 T2886 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1034819 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1034819 T2919 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1034819 T2919 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1016 name=collection1 org.apache.solr.core.SolrCore@74485d5f url=https://127.0.0.1:34510/collection1 node=127.0.0.1:34510_ C1016_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:34510, node_name=127.0.0.1:34510_, state=down}
   [junit4]   2> 1034819 T2919 C1016 P34510 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:34510/collection1/
   [junit4]   2> 1034820 T2919 C1016 P34510 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1034820 T2886 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1034820 T2919 C1016 P34510 oasc.SyncStrategy.syncToMe https://127.0.0.1:34510/collection1/ has no replicas
   [junit4]   2> 1034820 T2919 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:34510/collection1/ shard1
   [junit4]   2> 1034820 T2886 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1034820 T2919 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1034820 T2886 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1034821 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1034826 T2923 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4925023 name:ZooKeeperConnection Watcher:127.0.0.1:49267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1034826 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1034826 T2886 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1034831 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1034833 T2911 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: collection1
   [junit4]   2> 1034834 T2911 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 1034834 T2911 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 2
   [junit4]   2> 1034835 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1034835 T2911 oasc.Overseer$ClusterStateUpdater.updateZkStates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1034836 T2915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1034837 T2924 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1034890 T2919 oasc.ZkController.register We are https://127.0.0.1:34510/collection1/ and leader is https://127.0.0.1:34510/collection1/
   [junit4]   2> 1034890 T2919 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:34510
   [junit4]   2> 1034891 T2919 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1034891 T2919 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1034891 T2919 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1034899 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1034902 T2911 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:34510",
   [junit4]   2> 	  "node_name":"127.0.0.1:34510_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1035004 T2924 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1035004 T2915 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1035159 T2886 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1035160 T2886 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1035162 T2886 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1035164 T2886 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:36325
   [junit4]   2> 1035166 T2886 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1035166 T2886 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1035166 T2886 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003
   [junit4]   2> 1035167 T2886 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003/'
   [junit4]   2> 1035184 T2886 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003/solr.xml
   [junit4]   2> 1035193 T2886 oasc.CoreContainer.<init> New CoreContainer 426468420
   [junit4]   2> 1035193 T2886 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003/]
   [junit4]   2> 1035194 T2886 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1035194 T2886 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1035195 T2886 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1035195 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1035195 T2886 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1035195 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1035196 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1035196 T2886 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1035196 T2886 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1035198 T2886 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1035199 T2886 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1035199 T2886 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1035200 T2886 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1035200 T2886 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49267/solr
   [junit4]   2> 1035200 T2886 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1035201 T2886 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1035201 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1035207 T2935 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46b78e8d name:ZooKeeperConnection Watcher:127.0.0.1:49267 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1035208 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1035209 T2886 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1035211 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1035213 T2937 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2057d934 name:ZooKeeperConnection Watcher:127.0.0.1:49267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1035213 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1035217 T2886 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1036220 T2886 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36325_
   [junit4]   2> 1036221 T2886 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36325_
   [junit4]   2> 1036224 T2886 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1036228 T2939 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1036228 T2939 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1036229 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1036229 T2939 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1036229 T2939 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1036230 T2939 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1036230 T2911 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36325",
   [junit4]   2> 	  "node_name":"127.0.0.1:36325_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1036230 T2939 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 1036231 T2939 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1036232 T2911 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1036232 T2911 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1036234 T2911 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1036235 T2938 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1036235 T2938 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 1037231 T2939 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1037231 T2939 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1037232 T2939 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1037232 T2939 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1037233 T2939 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1037233 T2939 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003/collection1/'
   [junit4]   2> 1037233 T2939 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1037233 T2939 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1037234 T2939 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1037249 T2939 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1037257 T2939 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1037258 T2939 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1037261 T2939 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1037340 T2939 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1037343 T2939 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1037344 T2939 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1037349 T2939 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1037351 T2939 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1037352 T2939 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1037353 T2939 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1037354 T2939 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1037354 T2939 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1037355 T2939 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1037355 T2939 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1037355 T2939 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1037355 T2939 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1037355 T2939 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty1/
   [junit4]   2> 1037355 T2939 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f48baf
   [junit4]   2> 1037356 T2939 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty1
   [junit4]   2> 1037356 T2939 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty1/index/
   [junit4]   2> 1037356 T2939 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1037356 T2939 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty1/index
   [junit4]   2> 1037356 T2939 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1037357 T2939 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-159af10a)),segFN=segments_1,generation=1}
   [junit4]   2> 1037357 T2939 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1037359 T2939 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1037359 T2939 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1037359 T2939 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1037359 T2939 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1037359 T2939 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1037359 T2939 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1037360 T2939 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1037360 T2939 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1037360 T2939 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1037360 T2939 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1037360 T2939 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1037361 T2939 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1037361 T2939 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1037361 T2939 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1037361 T2939 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1037361 T2939 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1037362 T2939 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 1037362 T2939 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 1037362 T2939 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1037362 T2939 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 1037364 T2939 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1037364 T2939 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1037365 T2939 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1037365 T2939 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1037366 T2939 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1037366 T2939 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1037367 T2939 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=82.064453125, floorSegmentMB=2.1884765625, forceMergeDeletesPctAllowed=17.518579599039594, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.34514987743752135
   [junit4]   2> 1037367 T2939 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-159af10a)),segFN=segments_1,generation=1}
   [junit4]   2> 1037367 T2939 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1037367 T2939 oass.SolrIndexSearcher.<init> Opening Searcher@4ddbea43[collection1] main
   [junit4]   2> 1037367 T2939 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1037368 T2939 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1037369 T2939 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1037369 T2939 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1037369 T2939 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1037369 T2939 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1037370 T2939 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1037370 T2939 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1037370 T2939 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 1037370 T2939 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1037371 T2940 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4ddbea43[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1037373 T2939 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1037373 T2943 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:36325 collection:collection1 shard:shard2
   [junit4]   2> 1037374 T2886 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1037374 T2943 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1037374 T2886 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1037376 T2943 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1037377 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1037377 T2943 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1037378 T2943 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1017 name=collection1 org.apache.solr.core.SolrCore@1dff6538 url=https://127.0.0.1:36325/collection1 node=127.0.0.1:36325_ C1017_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:36325, node_name=127.0.0.1:36325_, state=down}
   [junit4]   2> 1037378 T2943 C1017 P36325 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:36325/collection1/
   [junit4]   2> 1037378 T2943 C1017 P36325 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1037378 T2943 C1017 P36325 oasc.SyncStrategy.syncToMe https://127.0.0.1:36325/collection1/ has no replicas
   [junit4]   2> 1037378 T2943 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:36325/collection1/ shard2
   [junit4]   2> 1037378 T2943 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1037380 T2911 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1037380 T2938 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1037381 T2938 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 1037383 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1037383 T2911 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1037384 T2938 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1037384 T2938 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 1037431 T2943 oasc.ZkController.register We are https://127.0.0.1:36325/collection1/ and leader is https://127.0.0.1:36325/collection1/
   [junit4]   2> 1037431 T2943 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:36325
   [junit4]   2> 1037431 T2943 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1037432 T2943 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1037432 T2943 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1037433 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1037434 T2911 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:36325",
   [junit4]   2> 	  "node_name":"127.0.0.1:36325_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1037435 T2943 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 1037437 T2911 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1037437 T2938 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1037438 T2938 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 1037598 T2886 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1037599 T2886 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 1037602 T2886 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 1037603 T2886 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:51015
   [junit4]   2> 1037605 T2886 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 1037605 T2886 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1037605 T2886 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004
   [junit4]   2> 1037605 T2886 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004/'
   [junit4]   2> 1037614 T2886 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004/solr.xml
   [junit4]   2> 1037620 T2886 oasc.CoreContainer.<init> New CoreContainer 1513032300
   [junit4]   2> 1037621 T2886 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004/]
   [junit4]   2> 1037621 T2886 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1037621 T2886 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1037621 T2886 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1037622 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1037622 T2886 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1037622 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1037622 T2886 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1037622 T2886 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1037623 T2886 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1037624 T2886 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 1037624 T2886 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1037625 T2886 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1037625 T2886 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1037625 T2886 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49267/solr
   [junit4]   2> 1037625 T2886 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1037625 T2886 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1037626 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1037628 T2954 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c4a98e name:ZooKeeperConnection Watcher:127.0.0.1:49267 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1037629 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1037629 T2886 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1037631 T2886 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1037633 T2956 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a898a68 name:ZooKeeperConnection Watcher:127.0.0.1:49267/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1037633 T2886 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1037636 T2886 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1038639 T2886 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51015_
   [junit4]   2> 1038640 T2886 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51015_
   [junit4]   2> 1038642 T2886 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1038646 T2958 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1038646 T2958 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1038647 T2915 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 1038648 T2958 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1038648 T2958 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1038648 T2958 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1038649 T2911 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:51015",
   [junit4]   2> 	  "node_name":"127.0.0.1:51015_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1038649 T2958 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 1038649 T2958 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1038650 T2911 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 1038650 T2911 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1038652 T2911 oasc.Overseer$ClusterStateUpdater.updateZkStates going to update_collection /collections/collection1/state.json
   [junit4]   2> 1038653 T2938 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1038653 T2957 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1038653 T2938 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 1038653 T2957 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 1039649 T2958 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1039650 T2958 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1039650 T2958 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1039650 T2958 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1039651 T2958 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1039651 T2958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004/collection1/'
   [junit4]   2> 1039651 T2958 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1039652 T2958 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1039652 T2958 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1039668 T2958 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1039676 T2958 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1039677 T2958 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 1039680 T2958 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1039757 T2958 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1039760 T2958 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1039761 T2958 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1039767 T2958 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1039768 T2958 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1039769 T2958 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1039770 T2958 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1039771 T2958 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1039771 T2958 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1039771 T2958 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1039772 T2958 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1039772 T2958 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1039772 T2958 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1039772 T2958 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty2/
   [junit4]   2> 1039772 T2958 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2f48baf
   [junit4]   2> 1039773 T2958 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty2
   [junit4]   2> 1039773 T2958 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty2/index/
   [junit4]   2> 1039773 T2958 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1039773 T2958 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-48532332BF07798A-001/tempDir-001/jetty2/index
   [junit4]   2> 1039773 T2958 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 1039774 T2958 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-71005bba)),segFN=segments_1,generation=1}
   [junit4]   2> 1039774 T2958 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1039775 T2958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1039775 T2958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1039776 T2958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1039776 T2958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1039776 T2958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1039776 T2958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1039776 T2958 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1039776 T2958 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1039776 T2958 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1039777 T2958 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1039777 T2958 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1039777 T2958 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1039777 T2958 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1039778 T2958 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 1039778 T2958 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 1039778 T2958 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 1039778 T2958 oasc.RequestHandlers.initHandlersFromConfig created /updat

[...truncated too long message...]

e:doc*&indent=true&timeAllowed=1} status=400 QTime=3 
   [junit4]   2> 2482566 T7092 oas.SolrTestCaseJ4.tearDown ###Ending testQueriesOnDocsWithMultipleTerms
   [junit4]   2> 2482569 T7092 oas.SolrTestCaseJ4.setUp ###Starting testPrefixQuery
   [junit4]   2> 2482570 T7092 C1192 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=name:a*&indent=true&timeAllowed=1} hits=1999 status=0 QTime=1 
   [junit4]   2> 2482570 T7092 oas.SolrTestCaseJ4.tearDown ###Ending testPrefixQuery
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ExitableDirectoryReaderTest -Dtests.method=testPrefixQuery -Dtests.seed=48532332BF07798A -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_HK_#Hant -Dtests.timezone=Asia/Kuala_Lumpur -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 0.01s J1 | ExitableDirectoryReaderTest.testPrefixQuery <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: 
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([48532332BF07798A:FB041770DBDA9D0F]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQEx(SolrTestCaseJ4.java:850)
   [junit4]    > 	at org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery(ExitableDirectoryReaderTest.java:61)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2482574 T7092 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 2482574 T7092 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=649285765
   [junit4]   2> 2482574 T7092 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@5c7cd3a4
   [junit4]   2> 2482574 T7092 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5997,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 2482575 T7092 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 2482575 T7092 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2482575 T7092 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2482575 T7092 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2482575 T7092 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2482576 T7092 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-48532332BF07798A-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-48532332BF07798A-001/init-core-data-001;done=false>>]
   [junit4]   2> 2482576 T7092 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-48532332BF07798A-001/init-core-data-001
   [junit4]   2> 2482576 T7092 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-48532332BF07798A-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-48532332BF07798A-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 2482576 T7092 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-48532332BF07798A-001/init-core-data-001/index
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-48532332BF07798A-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {name=PostingsFormat(name=Lucene41VarGapFixedInterval), id=PostingsFormat(name=Lucene41VarGapFixedInterval)}, docValues:{}, sim=DefaultSimilarity, locale=zh_HK_#Hant, timezone=Asia/Kuala_Lumpur
   [junit4]   2> NOTE: Linux 3.13.0-36-generic amd64/Oracle Corporation 1.9.0-ea (64-bit)/cpus=8,threads=1,free=145240976,total=419430400
   [junit4]   2> NOTE: All tests run in this JVM: [UniqFieldsUpdateProcessorFactoryTest, TestIndexSearcher, TestChildDocTransformer, TestHighFrequencyDictionaryFactory, TestFieldCollectionResource, ResponseHeaderTest, TestLeaderElectionZkExpiry, SortByFunctionTest, AnalyticsMergeStrategyTest, TermVectorComponentDistributedTest, SoftAutoCommitTest, VMParamsZkACLAndCredentialsProvidersTest, TestIndexingPerformance, TestFileDictionaryLookup, AddSchemaFieldsUpdateProcessorFactoryTest, LeaderFailoverAfterPartitionTest, AutoCommitTest, TestMaxScoreQueryParser, QueryElevationComponentTest, OpenCloseCoreStressTest, AliasIntegrationTest, TestJmxMonitoredMap, TestValueSourceCache, ScriptEngineTest, TestStandardQParsers, TestManagedResourceStorage, SpatialFilterTest, ReplicationFactorTest, TestShortCircuitedRequests, DeleteInactiveReplicaTest, TestNoOpRegenerator, AddBlockUpdateTest, TestCSVResponseWriter, TestManagedSchemaDynamicFieldResource, TestFieldTypeCollectionResource, TestRealTimeGet, SampleTest, CoreMergeIndexesAdminHandlerTest, TestQuerySenderNoQuery, TestWriterPerf, AssignTest, TestInfoStreamLogging, BlockDirectoryTest, HdfsLockFactoryTest, TestCollationField, SolrInfoMBeanTest, TestDistributedGrouping, TestAtomicUpdateErrorCases, ZkSolrClientTest, TestRandomFaceting, SSLMigrationTest, TestImplicitCoreProperties, TestDistribDocBasedVersion, ShardRoutingCustomTest, TestMergePolicyConfig, CoreAdminRequestStatusTest, SchemaVersionSpecificBehaviorTest, SolrCoreTest, LegacyHTMLStripCharFilterTest, TestLFUCache, CSVRequestHandlerTest, SearchHandlerTest, DocValuesTest, TestGroupingSearch, URLClassifyProcessorTest, TestNRTOpen, TestCopyFieldCollectionResource, TestPerFieldSimilarity, TestSortingResponseWriter, TestSolrXmlPersistor, TestReversedWildcardFilterFactory, TestCSVLoader, FullSolrCloudDistribCmdsTest, TestDynamicFieldCollectionResource, ClusterStateTest, TestIBSimilarityFactory, TestQueryTypes, CopyFieldTest, TestAddFieldRealTimeGet, PreAnalyzedFieldTest, TestCursorMarkWithoutUniqueKey, CacheHeaderTest, CustomCollectionTest, SuggesterFSTTest, SharedFSAutoReplicaFailoverTest, TestFiltering, NumericFieldsTest, DistributedQueryComponentOptimizationTest, SynonymTokenizerTest, RequestHandlersTest, TestCoreContainer, StressHdfsTest, OutputWriterTest, TestPostingsSolrHighlighter, QueryResultKeyTest, SliceStateTest, BasicZkTest, AnalysisAfterCoreReloadTest, TestUpdate, CurrencyFieldXmlFileTest, DistributedQueryElevationComponentTest, HighlighterMaxOffsetTest, TestFunctionQuery, DeleteShardTest, TestCharFilters, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, LeaderElectionIntegrationTest, TestReplicationHandler, SolrCoreCheckLockOnStartupTest, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, DisMaxRequestHandlerTest, TestQueryUtils, DirectSolrSpellCheckerTest, FileBasedSpellCheckerTest, DocumentBuilderTest, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, ReturnFieldsTest, JSONWriterTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, PingRequestHandlerTest, TestBinaryResponseWriter, TestQuerySenderListener, AlternateDirectoryTest, SolrIndexConfigTest, TestSolrDeletionPolicy2, TestDocSet, TestElisionMultitermQuery, TestConfig, TestSolrCoreProperties, TestXIncludeConfig, TestLMDirichletSimilarityFactory, TestDefaultSimilarityFactory, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, SolrTestCaseJ4Test, TestHighlightDedupGrouping, TestTolerantSearch, CollectionsAPIAsyncDistributedZkTest, DeleteReplicaTest, RollingRestartTest, SharedFSAutoReplicaFailoverUtilsTest, SolrXmlInZkTest, TestCloudPivotFacet, TestCollectionAPI, TestModifyConfFiles, TestRequestStatusCollectionAPI, HdfsBasicDistributedZk2Test, HdfsChaosMonkeySafeLeaderTest, HdfsCollectionsAPIDistributedZkTest, HdfsRecoveryZkTest, HdfsSyncSliceTest, ExitableDirectoryReaderTest]
   [junit4] Completed on J1 in 2.67s, 2 tests, 1 failure <<< FAILURES!

[...truncated 196 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:524: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:472: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1359: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 1 error, 2 failures, 44 ignored (22 assumptions)

Total time: 92 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any