You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/09/25 19:18:49 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b28) - Build # 11328 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11328/
Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Send doc 2 to old leader core_node2 should have failed! ClusterState: DocCollection(c8n_1x2_leader_session_loss)={   "replicationFactor":"2",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "core":"c8n_1x2_leader_session_loss_shard1_replica1",           "base_url":"http://127.0.0.1:51727/_q/jh",           "node_name":"127.0.0.1:51727__q%2Fjh",           "state":"active",           "leader":"true"},         "core_node2":{           "core":"c8n_1x2_leader_session_loss_shard1_replica2",           "base_url":"http://127.0.0.1:38917/_q/jh",           "node_name":"127.0.0.1:38917__q%2Fjh",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Send doc 2 to old leader core_node2 should have failed! ClusterState: DocCollection(c8n_1x2_leader_session_loss)={
  "replicationFactor":"2",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "core":"c8n_1x2_leader_session_loss_shard1_replica1",
          "base_url":"http://127.0.0.1:51727/_q/jh",
          "node_name":"127.0.0.1:51727__q%2Fjh",
          "state":"active",
          "leader":"true"},
        "core_node2":{
          "core":"c8n_1x2_leader_session_loss_shard1_replica2",
          "base_url":"http://127.0.0.1:38917/_q/jh",
          "node_name":"127.0.0.1:38917__q%2Fjh",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
	at __randomizedtesting.SeedInfo.seed([392D1ED40D5A9101:B8CB90CC7A05F13D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.HttpPartitionTest.testLeaderZkSessionLoss(HttpPartitionTest.java:329)
	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:119)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:484)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11982 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/init-core-data-001
   [junit4]   2> 2291287 T6571 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_q/jh
   [junit4]   2> 2291290 T6571 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2291291 T6571 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2291291 T6572 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2291391 T6571 oasc.ZkTestServer.run start zk server on port:46769
   [junit4]   2> 2291392 T6571 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2291392 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2291394 T6578 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@508b74a3 name:ZooKeeperConnection Watcher:127.0.0.1:46769 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2291394 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2291395 T6571 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2291395 T6571 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2291398 T6571 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2291399 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2291400 T6580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c2b8367 name:ZooKeeperConnection Watcher:127.0.0.1:46769/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2291400 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2291400 T6571 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2291400 T6571 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2291403 T6571 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2291404 T6571 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2291405 T6571 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2291406 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2291407 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2291408 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2291409 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2291410 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2291411 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2291412 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2291413 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2291414 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2291414 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2291416 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2291416 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2291417 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2291418 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2291419 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2291419 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2291421 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2291421 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2291422 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2291423 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2291424 T6571 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2291425 T6571 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2291811 T6571 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2291813 T6571 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48708
   [junit4]   2> 2291814 T6571 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2291814 T6571 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2291814 T6571 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002
   [junit4]   2> 2291815 T6571 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002/'
   [junit4]   2> 2291828 T6571 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002/solr.xml
   [junit4]   2> 2291837 T6571 oasc.CoreContainer.<init> New CoreContainer 1691369757
   [junit4]   2> 2291837 T6571 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002/]
   [junit4]   2> 2291838 T6571 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2291838 T6571 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2291839 T6571 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2291839 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2291839 T6571 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2291839 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2291840 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2291840 T6571 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2291840 T6571 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2291841 T6571 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2291841 T6571 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2291841 T6571 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2291842 T6571 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2291842 T6571 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46769/solr
   [junit4]   2> 2291842 T6571 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2291843 T6571 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2291843 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2291844 T6591 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61f25a43 name:ZooKeeperConnection Watcher:127.0.0.1:46769 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2291845 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2291845 T6571 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2291847 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2291847 T6593 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16f01310 name:ZooKeeperConnection Watcher:127.0.0.1:46769/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2291848 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2291849 T6571 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2291850 T6571 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2291852 T6571 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2291853 T6571 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2291854 T6571 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2291856 T6571 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2291857 T6571 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51727__q%2Fjh
   [junit4]   2> 2291857 T6571 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51727__q%2Fjh
   [junit4]   2> 2291858 T6571 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2291859 T6571 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2291860 T6571 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2291861 T6571 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51727__q%2Fjh
   [junit4]   2> 2291862 T6571 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2291863 T6571 oasc.Overseer.start Overseer (id=92514884088627203-127.0.0.1:51727__q%2Fjh-n_0000000000) starting
   [junit4]   2> 2291865 T6571 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2291869 T6571 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2291870 T6595 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2291870 T6571 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2291872 T6571 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2291873 T6571 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2291875 T6594 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2291876 T6597 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2291877 T6597 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2291877 T6597 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2291877 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2291878 T6594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51727/_q/jh",
   [junit4]   2> 	  "node_name":"127.0.0.1:51727__q%2Fjh",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2291879 T6594 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2291879 T6594 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 2291879 T6594 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2291880 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2292877 T6597 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2292878 T6597 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2292878 T6597 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2292878 T6597 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2292879 T6597 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2292879 T6597 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002/collection1/'
   [junit4]   2> 2292880 T6597 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2292880 T6597 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2292880 T6597 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2292904 T6597 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2292912 T6597 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2292913 T6597 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2292916 T6597 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2292985 T6597 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2292988 T6597 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2292989 T6597 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2292995 T6597 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2292996 T6597 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2292997 T6597 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2292998 T6597 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2292998 T6597 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2292998 T6597 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2292998 T6597 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2292998 T6597 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2292999 T6597 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2292999 T6597 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2292999 T6597 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/control/data/
   [junit4]   2> 2292999 T6597 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@21f15510
   [junit4]   2> 2292999 T6597 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/control/data
   [junit4]   2> 2293000 T6597 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/control/data/index/
   [junit4]   2> 2293000 T6597 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2293000 T6597 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/control/data/index
   [junit4]   2> 2293000 T6597 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2293001 T6597 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-5bb83d56)),segFN=segments_1,generation=1}
   [junit4]   2> 2293001 T6597 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2293002 T6597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2293002 T6597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2293002 T6597 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2293003 T6597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2293003 T6597 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2293003 T6597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2293003 T6597 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2293003 T6597 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2293003 T6597 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2293004 T6597 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2293004 T6597 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2293004 T6597 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2293004 T6597 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2293004 T6597 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2293005 T6597 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2293005 T6597 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2293005 T6597 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2293005 T6597 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2293005 T6597 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2293005 T6597 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2293007 T6597 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2293007 T6597 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2293008 T6597 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2293008 T6597 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2293009 T6597 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2293010 T6597 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2293010 T6597 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1968472227, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2293010 T6597 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-5bb83d56)),segFN=segments_1,generation=1}
   [junit4]   2> 2293010 T6597 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2293011 T6597 oass.SolrIndexSearcher.<init> Opening Searcher@47dfac5d[collection1] main
   [junit4]   2> 2293011 T6597 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2293011 T6597 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2293012 T6597 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2293012 T6597 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2293012 T6597 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2293012 T6597 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2293013 T6597 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2293013 T6597 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2293013 T6597 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2293014 T6597 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2293014 T6597 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2293014 T6597 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2293016 T6599 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47dfac5d[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2293016 T6597 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2293022 T6602 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51727/_q/jh collection:control_collection shard:shard1
   [junit4]   2> 2293022 T6571 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2293022 T6571 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2293023 T6602 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2293025 T6571 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2293025 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2293027 T6605 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@247cad6d name:ZooKeeperConnection Watcher:127.0.0.1:46769/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2293027 T6602 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2293027 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2293028 T6571 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2293028 T6571 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2293030 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2293030 T6602 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2293030 T6571 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2293030 T6602 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3201 name=collection1 org.apache.solr.core.SolrCore@6cf9223e url=http://127.0.0.1:51727/_q/jh/collection1 node=127.0.0.1:51727__q%2Fjh C3201_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:51727/_q/jh, node_name=127.0.0.1:51727__q%2Fjh, state=down}
   [junit4]   2> 2293031 T6602 C3201 P51727 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51727/_q/jh/collection1/
   [junit4]   2> 2293031 T6602 C3201 P51727 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2293031 T6602 C3201 P51727 oasc.SyncStrategy.syncToMe http://127.0.0.1:51727/_q/jh/collection1/ has no replicas
   [junit4]   2> 2293031 T6602 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51727/_q/jh/collection1/ shard1
   [junit4]   2> 2293031 T6602 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2293038 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2293139 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2293140 T6606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2293188 T6602 oasc.ZkController.register We are http://127.0.0.1:51727/_q/jh/collection1/ and leader is http://127.0.0.1:51727/_q/jh/collection1/
   [junit4]   2> 2293188 T6602 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51727/_q/jh
   [junit4]   2> 2293189 T6602 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2293189 T6602 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2293189 T6602 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2293190 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2293192 T6594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51727/_q/jh",
   [junit4]   2> 	  "node_name":"127.0.0.1:51727__q%2Fjh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2293270 T6571 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2293271 T6571 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2293273 T6571 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51165
   [junit4]   2> 2293274 T6571 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2293274 T6571 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2293274 T6571 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003
   [junit4]   2> 2293275 T6571 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003/'
   [junit4]   2> 2293284 T6571 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003/solr.xml
   [junit4]   2> 2293290 T6571 oasc.CoreContainer.<init> New CoreContainer 1898320971
   [junit4]   2> 2293290 T6571 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003/]
   [junit4]   2> 2293291 T6571 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2293291 T6571 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2293291 T6571 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2293291 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2293292 T6571 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2293292 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2293292 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2293292 T6571 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2293292 T6571 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2293293 T6571 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2293293 T6571 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2293293 T6571 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2293294 T6606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2293294 T6571 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2293294 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2293294 T6571 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46769/solr
   [junit4]   2> 2293296 T6571 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2293296 T6571 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2293299 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2293300 T6617 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69efc7fe name:ZooKeeperConnection Watcher:127.0.0.1:46769 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2293300 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2293300 T6571 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2293301 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2293302 T6619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b009b83 name:ZooKeeperConnection Watcher:127.0.0.1:46769/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2293302 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2293305 T6571 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2294307 T6571 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38917__q%2Fjh
   [junit4]   2> 2294308 T6571 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38917__q%2Fjh
   [junit4]   2> 2294317 T6571 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2294320 T6621 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2294320 T6621 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2294321 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2294321 T6621 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2294323 T6594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38917/_q/jh",
   [junit4]   2> 	  "node_name":"127.0.0.1:38917__q%2Fjh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2294323 T6594 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2294323 T6594 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
   [junit4]   2> 2294323 T6594 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2294425 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2294425 T6620 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2294425 T6606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2295321 T6621 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2295321 T6621 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2295322 T6621 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2295322 T6621 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2295323 T6621 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2295323 T6621 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003/collection1/'
   [junit4]   2> 2295323 T6621 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2295323 T6621 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2295324 T6621 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2295339 T6621 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2295346 T6621 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2295347 T6621 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2295350 T6621 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2295426 T6621 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2295429 T6621 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2295430 T6621 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2295436 T6621 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2295438 T6621 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2295439 T6621 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2295439 T6621 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2295440 T6621 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2295441 T6621 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2295441 T6621 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2295441 T6621 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2295442 T6621 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2295442 T6621 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2295442 T6621 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty1/
   [junit4]   2> 2295442 T6621 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@21f15510
   [junit4]   2> 2295442 T6621 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty1
   [junit4]   2> 2295443 T6621 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty1/index/
   [junit4]   2> 2295443 T6621 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2295443 T6621 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty1/index
   [junit4]   2> 2295443 T6621 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2295443 T6621 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-1e97281e)),segFN=segments_1,generation=1}
   [junit4]   2> 2295444 T6621 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2295445 T6621 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2295445 T6621 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2295446 T6621 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2295446 T6621 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2295446 T6621 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2295446 T6621 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2295446 T6621 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2295446 T6621 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2295446 T6621 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2295447 T6621 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2295447 T6621 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2295447 T6621 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2295448 T6621 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2295448 T6621 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2295448 T6621 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2295448 T6621 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2295448 T6621 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2295448 T6621 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2295449 T6621 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2295449 T6621 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2295450 T6621 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2295451 T6621 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2295451 T6621 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2295452 T6621 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2295454 T6621 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2295454 T6621 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2295456 T6621 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1968472227, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2295456 T6621 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-1e97281e)),segFN=segments_1,generation=1}
   [junit4]   2> 2295456 T6621 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2295456 T6621 oass.SolrIndexSearcher.<init> Opening Searcher@47bdaf4a[collection1] main
   [junit4]   2> 2295456 T6621 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2295457 T6621 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2295457 T6621 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2295458 T6621 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2295458 T6621 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2295458 T6621 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2295459 T6621 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2295459 T6621 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2295459 T6621 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2295459 T6621 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2295461 T6622 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@47bdaf4a[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2295462 T6621 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2295462 T6625 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38917/_q/jh collection:collection1 shard:shard2
   [junit4]   2> 2295463 T6571 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2295463 T6625 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2295463 T6571 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2295465 T6625 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2295466 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2295466 T6625 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2295466 T6625 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3202 name=collection1 org.apache.solr.core.SolrCore@6723d273 url=http://127.0.0.1:38917/_q/jh/collection1 node=127.0.0.1:38917__q%2Fjh C3202_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:38917/_q/jh, node_name=127.0.0.1:38917__q%2Fjh, state=down}
   [junit4]   2> 2295467 T6625 C3202 P38917 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38917/_q/jh/collection1/
   [junit4]   2> 2295467 T6625 C3202 P38917 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2295467 T6625 C3202 P38917 oasc.SyncStrategy.syncToMe http://127.0.0.1:38917/_q/jh/collection1/ has no replicas
   [junit4]   2> 2295467 T6625 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38917/_q/jh/collection1/ shard2
   [junit4]   2> 2295467 T6625 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2295469 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2295572 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2295572 T6620 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2295572 T6606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2295620 T6625 oasc.ZkController.register We are http://127.0.0.1:38917/_q/jh/collection1/ and leader is http://127.0.0.1:38917/_q/jh/collection1/
   [junit4]   2> 2295621 T6625 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38917/_q/jh
   [junit4]   2> 2295621 T6625 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2295621 T6625 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2295621 T6625 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2295622 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2295624 T6594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:38917/_q/jh",
   [junit4]   2> 	  "node_name":"127.0.0.1:38917__q%2Fjh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2295700 T6571 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2295700 T6571 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2295702 T6571 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:34067
   [junit4]   2> 2295703 T6571 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2295703 T6571 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2295704 T6571 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004
   [junit4]   2> 2295704 T6571 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004/'
   [junit4]   2> 2295714 T6571 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004/solr.xml
   [junit4]   2> 2295721 T6571 oasc.CoreContainer.<init> New CoreContainer 1560828598
   [junit4]   2> 2295721 T6571 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004/]
   [junit4]   2> 2295722 T6571 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2295722 T6571 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2295722 T6571 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2295722 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2295722 T6571 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2295723 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2295723 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2295723 T6571 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2295723 T6571 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2295724 T6571 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2295724 T6571 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2295724 T6571 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2295724 T6571 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2295725 T6571 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46769/solr
   [junit4]   2> 2295725 T6571 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2295725 T6571 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2295725 T6620 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2295726 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2295725 T6606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2295725 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2295728 T6637 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@733fe589 name:ZooKeeperConnection Watcher:127.0.0.1:46769 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2295728 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2295729 T6571 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2295730 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2295730 T6639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@214a5287 name:ZooKeeperConnection Watcher:127.0.0.1:46769/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2295730 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2295733 T6571 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2296735 T6571 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53905__q%2Fjh
   [junit4]   2> 2296736 T6571 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53905__q%2Fjh
   [junit4]   2> 2296739 T6571 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2296741 T6641 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2296741 T6641 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2296742 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2296742 T6641 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2296743 T6594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53905/_q/jh",
   [junit4]   2> 	  "node_name":"127.0.0.1:53905__q%2Fjh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2296744 T6594 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 2296744 T6594 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2296846 T6640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2296846 T6620 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2296846 T6606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2296846 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2297742 T6641 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2297743 T6641 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2297743 T6641 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2297743 T6641 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2297744 T6641 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2297744 T6641 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004/collection1/'
   [junit4]   2> 2297744 T6641 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2297745 T6641 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2297745 T6641 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2297761 T6641 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2297768 T6641 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2297769 T6641 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2297772 T6641 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2297849 T6641 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2297852 T6641 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2297853 T6641 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2297859 T6641 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2297860 T6641 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2297862 T6641 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2297862 T6641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2297863 T6641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2297864 T6641 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2297864 T6641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2297864 T6641 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2297865 T6641 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2297865 T6641 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2297865 T6641 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty2/
   [junit4]   2> 2297865 T6641 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@21f15510
   [junit4]   2> 2297865 T6641 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty2
   [junit4]   2> 2297866 T6641 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty2/index/
   [junit4]   2> 2297866 T6641 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2297866 T6641 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-001/jetty2/index
   [junit4]   2> 2297866 T6641 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2297866 T6641 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-519bc39b)),segFN=segments_1,generation=1}
   [junit4]   2> 2297867 T6641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2297868 T6641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2297869 T6641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2297869 T6641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2297869 T6641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2297869 T6641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2297869 T6641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2297869 T6641 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2297869 T6641 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2297870 T6641 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2297870 T6641 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2297870 T6641 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2297870 T6641 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2297871 T6641 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2297871 T6641 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2297871 T6641 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2297871 T6641 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2297871 T6641 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2297872 T6641 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2297872 T6641 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2297872 T6641 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2297873 T6641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2297874 T6641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2297875 T6641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2297875 T6641 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2297876 T6641 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2297876 T6641 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2297877 T6641 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1968472227, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2297877 T6641 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-519bc39b)),segFN=segments_1,generation=1}
   [junit4]   2> 2297877 T6641 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2297877 T6641 oass.SolrIndexSearcher.<init> Opening Searcher@39cfe7fe[collection1] main
   [junit4]   2> 2297878 T6641 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2297878 T6641 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2297878 T6641 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2297879 T6641 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2297879 T6641 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2297879 T6641 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2297879 T6641 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2297880 T6641 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2297880 T6641 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2297880 T6641 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2297882 T6642 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39cfe7fe[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2297883 T6641 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2297883 T6645 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53905/_q/jh collection:collection1 shard:shard1
   [junit4]   2> 2297883 T6571 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 2297884 T6645 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2297884 T6571 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2297886 T6645 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2297887 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2297887 T6645 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2297887 T6645 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C3203 name=collection1 org.apache.solr.core.SolrCore@12d5f718 url=http://127.0.0.1:53905/_q/jh/collection1 node=127.0.0.1:53905__q%2Fjh C3203_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:53905/_q/jh, node_name=127.0.0.1:53905__q%2Fjh, state=down}
   [junit4]   2> 2297887 T6645 C3203 P53905 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53905/_q/jh/collection1/
   [junit4]   2> 2297888 T6645 C3203 P53905 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2297888 T6645 C3203 P53905 oasc.SyncStrategy.syncToMe http://127.0.0.1:53905/_q/jh/collection1/ has no replicas
   [junit4]   2> 2297888 T6645 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53905/_q/jh/collection1/ shard1
   [junit4]   2> 2297888 T6645 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2297890 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2297993 T6640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2297993 T6620 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2297993 T6606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2297993 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2298041 T6645 oasc.ZkController.register We are http://127.0.0.1:53905/_q/jh/collection1/ and leader is http://127.0.0.1:53905/_q/jh/collection1/
   [junit4]   2> 2298041 T6645 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53905/_q/jh
   [junit4]   2> 2298042 T6645 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2298042 T6645 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2298042 T6645 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2298043 T6598 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2298045 T6594 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:53905/_q/jh",
   [junit4]   2> 	  "node_name":"127.0.0.1:53905__q%2Fjh",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2298126 T6571 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2298128 T6571 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2298129 T6571 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39019
   [junit4]   2> 2298131 T6571 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2298131 T6571 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2298131 T6571 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-005
   [junit4]   2> 2298131 T6571 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-005/'
   [junit4]   2> 2298141 T6571 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-005/solr.xml
   [junit4]   2> 2298147 T6571 oasc.CoreContainer.<init> New CoreContainer 1166841101
   [junit4]   2> 2298147 T6571 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001/tempDir-005/]
   [junit4]   2> 2298148 T6620 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2298148 T6571 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2298148 T6571 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2298148 T6571 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2298149 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2298148 T6640 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2298148 T6598 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2298148 T6606 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2298149 T6571 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2298150 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2298150 T6571 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2298150 T6571 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2298150 T6571 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2298151 T6571 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2298151 T6571 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2298151 T6571 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2298152 T6571 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2298152 T6571 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46769/solr
   [junit4]   2> 2298152 T6571 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2298152 T6571 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2298153 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2298154 T6657 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59997aec name:ZooKeeperConnection Watcher:127.0.0.1:46769 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2298154 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2298155 T6571 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2298156 T6571 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2298157 T6659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@184bbdb2 name:ZooKeeperConnection Watcher:127.0.0.1:46769/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2298157 T6571 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2298160 T6571 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2299162 T6571 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48847__q%2Fjh
   [junit4]   2> 2299163 T6571 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48847__q%2Fjh
   [junit4]   2> 2299166 T6571 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2299168 T6661 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2299168 T6661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2299169 T6598 oasc.DistributedQueue$LatchChildWatcher.process La

[...truncated too long message...]

dler.doStop stopped o.e.j.s.ServletContextHandler{/_q/jh,null}
   [junit4]   2> 2400816 T6571 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 2400817 T6571 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46769 46769
   [junit4]   2> 2401503 T6572 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:46769 46769
   [junit4]   2> 2401505 T6572 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		6	/solr/aliases.json
   [junit4]   2> 		6	/solr/clusterstate.json
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent data watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		2	/solr/overseer_elect/election/92514884088627203-127.0.0.1:51727__q%2Fjh-n_0000000000
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		7	/solr/live_nodes
   [junit4]   2> 		3	/solr/overseer/queue
   [junit4]   2> 		3	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> 2401505 T6571 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:53905/_q/jh
   [junit4]   2> 2401505 T6571 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:48847/_q/jh
   [junit4]   2> 2401506 T6571 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:38917/_q/jh
   [junit4]   2> 2401506 T6571 oasc.SocketProxy.close WARN Closing 0 connections to: http://127.0.0.1:51727/_q/jh
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HttpPartitionTest -Dtests.method=testDistribSearch -Dtests.seed=392D1ED40D5A9101 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en -Dtests.timezone=Africa/Addis_Ababa -Dtests.file.encoding=UTF-8
   [junit4] FAILURE  110s J1 | HttpPartitionTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Send doc 2 to old leader core_node2 should have failed! ClusterState: DocCollection(c8n_1x2_leader_session_loss)={
   [junit4]    >   "replicationFactor":"2",
   [junit4]    >   "shards":{"shard1":{
   [junit4]    >       "range":"80000000-7fffffff",
   [junit4]    >       "state":"active",
   [junit4]    >       "replicas":{
   [junit4]    >         "core_node1":{
   [junit4]    >           "core":"c8n_1x2_leader_session_loss_shard1_replica1",
   [junit4]    >           "base_url":"http://127.0.0.1:51727/_q/jh",
   [junit4]    >           "node_name":"127.0.0.1:51727__q%2Fjh",
   [junit4]    >           "state":"active",
   [junit4]    >           "leader":"true"},
   [junit4]    >         "core_node2":{
   [junit4]    >           "core":"c8n_1x2_leader_session_loss_shard1_replica2",
   [junit4]    >           "base_url":"http://127.0.0.1:38917/_q/jh",
   [junit4]    >           "node_name":"127.0.0.1:38917__q%2Fjh",
   [junit4]    >           "state":"down"}}}},
   [junit4]    >   "router":{"name":"compositeId"},
   [junit4]    >   "maxShardsPerNode":"1",
   [junit4]    >   "autoAddReplicas":"false"}
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([392D1ED40D5A9101:B8CB90CC7A05F13D]:0)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.testLeaderZkSessionLoss(HttpPartitionTest.java:329)
   [junit4]    > 	at org.apache.solr.cloud.HttpPartitionTest.doTest(HttpPartitionTest.java:119)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2401518 T6571 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.HttpPartitionTest-392D1ED40D5A9101-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {_version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), multiDefault=PostingsFormat(name=Lucene41VarGapFixedInterval), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), intDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), id=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=MockRandom), range_facet_l=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128))), timestamp=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene41(blocksize=128)))}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=en, timezone=Africa/Addis_Ababa
   [junit4]   2> NOTE: Linux 3.13.0-36-generic amd64/Oracle Corporation 1.9.0-ea (64-bit)/cpus=8,threads=1,free=64208704,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldCollectionResource, MultiTermTest, HdfsCollectionsAPIDistributedZkTest, ConnectionManagerTest, SuggestComponentTest, TestImplicitCoreProperties, SliceStateTest, TermVectorComponentTest, TestInitQParser, TestIndexingPerformance, TestTolerantSearch, AnalysisAfterCoreReloadTest, TestGroupingSearch, SolrPluginUtilsTest, FieldAnalysisRequestHandlerTest, TestBM25SimilarityFactory, TestFiltering, CSVRequestHandlerTest, TestValueSourceCache, TestManagedResourceStorage, TestInfoStreamLogging, TestCoreContainer, DateFieldTest, ParsingFieldUpdateProcessorsTest, UpdateRequestProcessorFactoryTest, RollingRestartTest, UniqFieldsUpdateProcessorFactoryTest, InfoHandlerTest, TestCSVResponseWriter, TestCloudManagedSchemaConcurrent, TestMultiCoreConfBootstrap, SampleTest, SearchHandlerTest, TestFieldTypeCollectionResource, TestAtomicUpdateErrorCases, TestMergePolicyConfig, TestChildDocTransformer, SolrXmlInZkTest, ShardSplitTest, TestDefaultSearchFieldResource, NumericFieldsTest, ShardRoutingCustomTest, DocValuesTest, TermVectorComponentDistributedTest, TestManagedSchemaFieldTypeResource, TestBadConfig, OpenCloseCoreStressTest, TestComplexPhraseQParserPlugin, TestRandomMergePolicy, TestIBSimilarityFactory, DistanceFunctionTest, VMParamsZkACLAndCredentialsProvidersTest, TestCopyFieldCollectionResource, CustomCollectionTest, TestUtils, FullSolrCloudDistribCmdsTest, DefaultValueUpdateProcessorTest, HardAutoCommitTest, FileBasedSpellCheckerTest, TestLeaderElectionZkExpiry, BlockDirectoryTest, MigrateRouteKeyTest, CursorPagingTest, SchemaVersionSpecificBehaviorTest, LoggingHandlerTest, TestQuerySenderNoQuery, TestDistribDocBasedVersion, TestCustomSort, HdfsLockFactoryTest, StatsComponentTest, AlternateDirectoryTest, CoreAdminRequestStatusTest, BasicZkTest, TestRequestStatusCollectionAPI, SpatialFilterTest, BasicDistributedZk2Test, TestCloudInspectUtil, RAMDirectoryFactoryTest, SolrIndexConfigTest, TestClassNameShortening, TestNRTOpen, TestCSVLoader, DistributedFacetPivotLongTailTest, AddBlockUpdateTest, DirectSolrConnectionTest, SynonymTokenizerTest, TestSchemaNameResource, TestUniqueKeyFieldResource, TestCloudSchemaless, BadCopyFieldTest, QueryElevationComponentTest, TestCollationField, SolrCoreTest, DateRangeFieldTest, SortByFunctionTest, SharedFSAutoReplicaFailoverTest, QueryResultKeyTest, TestCollectionAPI, TestIndexSearcher, TestReversedWildcardFilterFactory, TestManagedSchema, DirectUpdateHandlerOptimizeTest, ZkSolrClientTest, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest, TestReplicationHandler, TestFaceting, TestRecovery, TestHashPartitioner, TestJoin, DistributedTermsComponentTest, ConvertedLegacyTest, TestFunctionQuery, TestLazyCores, HighlighterTest, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestUpdate, TestWordDelimiterFilterFactory, TestRemoteStreaming, TestSurroundQueryParser, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestQueryUtils, TestWriterPerf, TestQueryTypes, TestOmitPositions, MoreLikeThisHandlerTest, TestSolrQueryParser, IndexSchemaRuntimeFieldTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, CoreContainerCoreInitFailuresTest, SOLR749Test, CopyFieldTest, TestStressRecovery, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, SpellingQueryConverterTest, ClusterStateTest, TestSolrJ, TestLRUCache, TestDocumentBuilder, ZkNodePropsTest, SystemInfoHandlerTest, UUIDFieldTest, FileUtilsTest, CircularListTest, TestRTGBase, DistributedIntervalFacetingTest, CollectionsAPIAsyncDistributedZkTest, DeleteInactiveReplicaTest, HttpPartitionTest]
   [junit4] Completed on J1 in 110.25s, 1 test, 1 failure <<< FAILURES!

[...truncated 214 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:524: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:472: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1359: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 430 suites, 1775 tests, 1 failure, 44 ignored (22 assumptions)

Total time: 87 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b28 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



Re: [JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20) - Build # 11329 - Still Failing!

Posted by Anshum Gupta <an...@anshumgupta.net>.
Looking at this. I'll increase the number of indexed documents for this
test for now so that this has really low chances of failing.
I'll also look up for other ways to test this that depends less on "let's
hope it takes long enough to hit the timeOut".

On Thu, Sep 25, 2014 at 3:51 PM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:

> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11329/
> Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseG1GC
>
> 3 tests failed.
> FAILED:
> org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch
>
> Error Message:
> no exception matching expected: 400: Request took too long during query
> expansion. Terminating request.
>
> Stack Trace:
> java.lang.AssertionError: no exception matching expected: 400: Request
> took too long during query expansion. Terminating request.
>         at
> __randomizedtesting.SeedInfo.seed([B805ADAE0F26DDD0:39E323B67879BDEC]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at
> org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
>         at
> org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81)
>         at
> org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
>         at
> org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:483)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> FAILED:
> org.apache.solr.core.ExitableDirectoryReaderTest.testQueriesOnDocsWithMultipleTerms
>
> Error Message:
>
>
> Stack Trace:
> java.lang.AssertionError:
>         at
> __randomizedtesting.SeedInfo.seed([B805ADAE0F26DDD0:D6C59E7F2353A6FF]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at
> org.apache.solr.SolrTestCaseJ4.assertQEx(SolrTestCaseJ4.java:850)
>         at
> org.apache.solr.core.ExitableDirectoryReaderTest.testQueriesOnDocsWithMultipleTerms(ExitableDirectoryReaderTest.java:88)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:483)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:745)
>
>
> FAILED:  org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery
>
> Error Message:
>
>
> Stack Trace:
> java.lang.AssertionError:
>         at
> __randomizedtesting.SeedInfo.seed([B805ADAE0F26DDD0:B5299EC6BFB3955]:0)
>         at org.junit.Assert.fail(Assert.java:93)
>         at
> org.apache.solr.SolrTestCaseJ4.assertQEx(SolrTestCaseJ4.java:850)
>         at
> org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery(ExitableDirectoryReaderTest.java:61)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:483)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
>         at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
>         at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
>         at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
>         at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
>         at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
>         at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
>         at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
>         at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>         at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> Build Log:
> [...truncated 11818 lines...]
>    [junit4] Suite: org.apache.solr.cloud.CloudExitableDirectoryReaderTest
>    [junit4]   2> Creating dataDir:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001
>    [junit4]   2> 2265251 T6489 oas.SolrTestCaseJ4.buildSSLConfig
> Randomized ssl (true) and clientAuth (true)
>    [junit4]   2> 2265251 T6489
> oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext
> system property: /
>    [junit4]   2> 2265255 T6489 oas.SolrTestCaseJ4.setUp ###Starting
> testDistribSearch
>    [junit4]   2> 2265255 T6489 oasc.ZkTestServer.run STARTING ZK TEST
> SERVER
>    [junit4]   1> client port:0.0.0.0/0.0.0.0:0
>    [junit4]   2> 2265256 T6490
> oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
>    [junit4]   2> 2265356 T6489 oasc.ZkTestServer.run start zk server on
> port:57711
>    [junit4]   2> 2265356 T6489
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 2265357 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2265360 T6496 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@3805495f
> name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2265360 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2265360 T6489 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
>    [junit4]   2> 2265360 T6489 oascc.SolrZkClient.makePath makePath: /solr
>    [junit4]   2> 2265363 T6489
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 2265366 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2265366 T6498 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@2fcbaf45
> name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2265367 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2265367 T6489 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
>    [junit4]   2> 2265367 T6489 oascc.SolrZkClient.makePath makePath:
> /collections/collection1
>    [junit4]   2> 2265368 T6489 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/shards
>    [junit4]   2> 2265370 T6489 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection
>    [junit4]   2> 2265371 T6489 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/shards
>    [junit4]   2> 2265372 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml
> to /configs/conf1/solrconfig.xml
>    [junit4]   2> 2265372 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.xml
>    [junit4]   2> 2265373 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml
> to /configs/conf1/schema.xml
>    [junit4]   2> 2265374 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/schema.xml
>    [junit4]   2> 2265375 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml
> to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 2265375 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/solrconfig.snippet.randomindexconfig.xml
>    [junit4]   2> 2265377 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt
> to /configs/conf1/stopwords.txt
>    [junit4]   2> 2265377 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/stopwords.txt
>    [junit4]   2> 2265378 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt
> to /configs/conf1/protwords.txt
>    [junit4]   2> 2265378 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/protwords.txt
>    [junit4]   2> 2265379 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml
> to /configs/conf1/currency.xml
>    [junit4]   2> 2265380 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/currency.xml
>    [junit4]   2> 2265381 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml
> to /configs/conf1/enumsConfig.xml
>    [junit4]   2> 2265381 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/enumsConfig.xml
>    [junit4]   2> 2265382 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json
> to /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 2265382 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/open-exchange-rates.json
>    [junit4]   2> 2265383 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt
> to /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 2265384 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/mapping-ISOLatin1Accent.txt
>    [junit4]   2> 2265385 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt
> to /configs/conf1/old_synonyms.txt
>    [junit4]   2> 2265385 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/old_synonyms.txt
>    [junit4]   2> 2265386 T6489 oasc.AbstractZkTestCase.putConfig put
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt
> to /configs/conf1/synonyms.txt
>    [junit4]   2> 2265386 T6489 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/synonyms.txt
>    [junit4]   2> 2265389 T6489
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 2265390 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2265391 T6500 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@346df4e9
> name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2265391 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2265391 T6489 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
>    [junit4]   2> 2265694 T6489 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 2265698 T6489 oejus.SslContextFactory.doStart Enabled
> Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello,
> SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>    [junit4]   2> 2265703 T6489 oejs.AbstractConnector.doStart Started
> SslSelectChannelConnector@127.0.0.1:58197
>    [junit4]   2> 2265705 T6489 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 2265705 T6489 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 2265705 T6489 oasc.SolrResourceLoader.locateSolrHome
> using system property solr.solr.home:
> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002
>    [junit4]   2> 2265706 T6489 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/'
>    [junit4]   2> 2265722 T6489 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/solr.xml
>    [junit4]   2> 2265733 T6489 oasc.CoreContainer.<init> New CoreContainer
> 1607842688
>    [junit4]   2> 2265733 T6489 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/]
>    [junit4]   2> 2265734 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 2265734 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 2265735 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 2265735 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 2265735 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 2265736 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 2265736 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 2265736 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 2265737 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 2265738 T6489 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 2265739 T6489 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 2265740 T6489 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 2265740 T6489 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 2265741 T6489 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:57711/solr
>    [junit4]   2> 2265741 T6489 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 2265741 T6489
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 2265742 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2265743 T6511 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@1ec8692c
> name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2265743 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2265743 T6489 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
>    [junit4]   2> 2265745 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2265746 T6513 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@6884a990
> name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2265746 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2265747 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer/queue
>    [junit4]   2> 2265748 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-queue-work
>    [junit4]   2> 2265750 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-running
>    [junit4]   2> 2265751 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-completed
>    [junit4]   2> 2265752 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer/collection-map-failure
>    [junit4]   2> 2265753 T6489 oascc.SolrZkClient.makePath makePath:
> /live_nodes
>    [junit4]   2> 2265754 T6489 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58197_
>    [junit4]   2> 2265754 T6489 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:58197_
>    [junit4]   2> 2265755 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer_elect
>    [junit4]   2> 2265756 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/election
>    [junit4]   2> 2265757 T6489 oasc.Overseer.close Overseer (id=null)
> closing
>    [junit4]   2> 2265758 T6489
> oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader
> 127.0.0.1:58197_
>    [junit4]   2> 2265758 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer_elect/leader
>    [junit4]   2> 2265759 T6489 oasc.Overseer.start Overseer
> (id=92516182783492100-127.0.0.1:58197_-n_0000000000) starting
>    [junit4]   2> 2265760 T6489 oascc.SolrZkClient.makePath makePath:
> /overseer/queue-work
>    [junit4]   2> 2265764 T6489
> oasc.OverseerAutoReplicaFailoverThread.<init> Starting
> OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000
> autoReplicaFailoverWaitAfterExpiration=30000
> autoReplicaFailoverBadNodeExpiration=60000
>    [junit4]   2> 2265764 T6515 oasc.OverseerCollectionProcessor.run
> Process current queue of collection creations
>    [junit4]   2> 2265764 T6489 oascc.SolrZkClient.makePath makePath:
> /clusterstate.json
>    [junit4]   2> 2265766 T6489 oascc.SolrZkClient.makePath makePath:
> /aliases.json
>    [junit4]   2> 2265766 T6489
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 2265768 T6514 oasc.Overseer$ClusterStateUpdater.run
> Starting to work on the main queue
>    [junit4]   2> 2265770 T6517 oasc.ZkController.publish publishing
> core=collection1 state=down collection=control_collection
>    [junit4]   2> 2265770 T6517 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 2265771 T6517 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 2265771 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2265772 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1
> message={
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "base_url":"https://127.0.0.1:58197",
>    [junit4]   2>          "node_name":"127.0.0.1:58197_",
>    [junit4]   2>          "numShards":"1",
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "collection":"control_collection",
>    [junit4]   2>          "operation":"state"}
>    [junit4]   2> 2265773 T6514
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> control_collection with shards [shard1]
>    [junit4]   2> 2265773 T6514
> oasc.Overseer$ClusterStateUpdater.createCollection state version
> control_collection 1
>    [junit4]   2> 2265773 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
>    [junit4]   2> 2265774 T6518 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 2266771 T6517 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 2266771 T6517 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:control_collection
>    [junit4]   2> 2266772 T6517 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 2266772 T6517 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/control_collection
>    [junit4]   2> 2266773 T6517 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
>    [junit4]   2> 2266773 T6517 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/'
>    [junit4]   2> 2266773 T6517 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 2266774 T6517 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 2266774 T6517 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/lib/README'
> to classloader
>    [junit4]   2> 2266791 T6517 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 2266844 T6517 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 2266845 T6517 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 2266850 T6517 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 2266951 T6517 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 2266955 T6517 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 2266956 T6517 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 2266962 T6517 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 2266964 T6517 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 2266965 T6517 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 2266966 T6517
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 2266966 T6517
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 2266966 T6517 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 2266967 T6517
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 2266967 T6517
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 2266967 T6517 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection control_collection
>    [junit4]   2> 2266967 T6517 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 2266967 T6517 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/,
> dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data/
>    [junit4]   2> 2266968 T6517 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1aff83b8
>    [junit4]   2> 2266968 T6517 oasc.CachingDirectoryFactory.get return new
> directory for
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data
>    [junit4]   2> 2266968 T6517 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data/index/
>    [junit4]   2> 2266968 T6517 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 2266969 T6517 oasc.CachingDirectoryFactory.get return new
> directory for
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data/index
>    [junit4]   2> 2266969 T6517 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 2266969 T6517 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-56990cda)),segFN=segments_1,generation=1}
>    [junit4]   2> 2266969 T6517 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 2266971 T6517 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 2266973 T6517 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 2266973 T6517 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 2266973 T6517 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 2266973 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2266973 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 2266975 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 2266975 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 2266975 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 2266975 T6517 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 2266995 T6517 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2266996 T6517 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2266996 T6517 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2266997 T6517 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2266998 T6517 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 2266998 T6517 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 2266999 T6517 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy:
> [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=25,
> maxMergedSegmentMB=14.1416015625, floorSegmentMB=1.9775390625,
> forceMergeDeletesPctAllowed=11.127099024208643, segmentsPerTier=45.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>    [junit4]   2> 2266999 T6517 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-56990cda)),segFN=segments_1,generation=1}
>    [junit4]   2> 2266999 T6517 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 2266999 T6517 oass.SolrIndexSearcher.<init> Opening
> Searcher@74a8b8af[collection1] main
>    [junit4]   2> 2266999 T6517 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/control_collection
>    [junit4]   2> 2267000 T6517 oascc.ZkStateReader.readConfigName
> path=/collections/control_collection configName=conf1 specified config
> exists in ZooKeeper
>    [junit4]   2> 2267000 T6517 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 2267001 T6517
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 2267001 T6517 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 2267001 T6517 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 2267001 T6517
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data
> found for znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 2267002 T6517 oasr.ManagedResourceStorage.load Loaded
> null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 2267002 T6517 oascc.SolrZkClient.makePath makePath:
> /configs/conf1/_rest_managed.json
>    [junit4]   2> 2267003 T6517
> oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to
> new znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 2267003 T6517
> oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 2267003 T6517 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 2267005 T6519 oasc.SolrCore.registerSearcher
> [collection1] Registered new searcher Searcher@74a8b8af[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 2267006 T6517 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 2267006 T6522 oasc.ZkController.register Register replica
> - core:collection1 address:https://127.0.0.1:58197
> collection:control_collection shard:shard1
>    [junit4]   2> 2267007 T6489 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
>    [junit4]   2> 2267007 T6489 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 2267008 T6522 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leader_elect/shard1/election
>    [junit4]   2> 2267009 T6489
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 2267010 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2267010 T6524 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@7bae41b3
> name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2267011 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2267011 T6489 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
>    [junit4]   2> 2267012 T6489
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 2267012 T6522
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard1
>    [junit4]   2> 2267013 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2267014 T6522
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
>    [junit4]   2> 2267014 T6489 oasc.ChaosMonkey.monkeyLog monkey: init -
> expire sessions:false cause connection loss:false
>    [junit4]   2> 2267014 T6522
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>    [junit4]   2> ASYNC  NEW_CORE C1657 name=collection1
> org.apache.solr.core.SolrCore@2cf5fa37 url=
> https://127.0.0.1:58197/collection1 node=127.0.0.1:58197_
> C1657_STATE=coll:control_collection core:collection1
> props:{core=collection1, base_url=https://127.0.0.1:58197,
> node_name=127.0.0.1:58197_, state=down}
>    [junit4]   2> 2267014 T6522 C1657 P58197 oasc.SyncStrategy.sync Sync
> replicas to https://127.0.0.1:58197/collection1/
>    [junit4]   2> 2267015 T6522 C1657 P58197 oasc.SyncStrategy.syncReplicas
> Sync Success - now sync replicas to me
>    [junit4]   2> 2267015 T6522 C1657 P58197 oasc.SyncStrategy.syncToMe
> https://127.0.0.1:58197/collection1/ has no replicas
>    [junit4]   2> 2267015 T6522
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> https://127.0.0.1:58197/collection1/ shard1
>    [junit4]   2> 2267015 T6522 oascc.SolrZkClient.makePath makePath:
> /collections/control_collection/leaders/shard1
>    [junit4]   2> 2267021 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2267022 T6518 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 2267022 T6525 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 2267070 T6522 oasc.ZkController.register We are
> https://127.0.0.1:58197/collection1/ and leader is
> https://127.0.0.1:58197/collection1/
>    [junit4]   2> 2267070 T6522 oasc.ZkController.register No LogReplay
> needed for core=collection1 baseURL=https://127.0.0.1:58197
>    [junit4]   2> 2267070 T6522 oasc.ZkController.checkRecovery I am the
> leader, no recovery necessary
>    [junit4]   2> 2267071 T6522 oasc.ZkController.publish publishing
> core=collection1 state=active collection=control_collection
>    [junit4]   2> 2267071 T6522 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 2267072 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2267074 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "core_node_name":"core_node1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "base_url":"https://127.0.0.1:58197",
>    [junit4]   2>          "node_name":"127.0.0.1:58197_",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "state":"active",
>    [junit4]   2>          "shard":"shard1",
>    [junit4]   2>          "collection":"control_collection",
>    [junit4]   2>          "operation":"state"}
>    [junit4]   2> 2267176 T6525 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 2267176 T6518 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
>    [junit4]   2> 2267278 T6489
> oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
>    [junit4]   2> 2267279 T6489 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 2267282 T6489 oejus.SslContextFactory.doStart Enabled
> Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello,
> SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>    [junit4]   2> 2267283 T6489 oejs.AbstractConnector.doStart Started
> SslSelectChannelConnector@127.0.0.1:42284
>    [junit4]   2> 2267285 T6489 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 2267286 T6489 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 2267286 T6489 oasc.SolrResourceLoader.locateSolrHome
> using system property solr.solr.home:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003
>    [junit4]   2> 2267286 T6489 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/'
>    [junit4]   2> 2267298 T6489 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/solr.xml
>    [junit4]   2> 2267307 T6489 oasc.CoreContainer.<init> New CoreContainer
> 1034069164
>    [junit4]   2> 2267307 T6489 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/]
>    [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 2267309 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 2267309 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 2267309 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 2267309 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 2267311 T6489 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 2267311 T6489 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 2267311 T6489 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 2267312 T6489 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 2267312 T6489 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:57711/solr
>    [junit4]   2> 2267312 T6489 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 2267313 T6489
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 2267313 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2267314 T6536 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5698aded
> name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2267314 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2267315 T6489 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
>    [junit4]   2> 2267316 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2267316 T6538 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@fbfe7d0
> name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2267316 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2267319 T6489
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 2268322 T6489 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42284_
>    [junit4]   2> 2268322 T6489 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:42284_
>    [junit4]   2> 2268325 T6489 oasc.Overseer.close Overseer (id=null)
> closing
>    [junit4]   2> 2268327 T6540 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 2268327 T6540 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 2268328 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2268328 T6540 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 2268329 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "base_url":"https://127.0.0.1:42284",
>    [junit4]   2>          "node_name":"127.0.0.1:42284_",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "operation":"state"}
>    [junit4]   2> 2268330 T6514
> oasc.Overseer$ClusterStateUpdater.createCollection Create collection
> collection1 with shards [shard1, shard2]
>    [junit4]   2> 2268330 T6514
> oasc.Overseer$ClusterStateUpdater.createCollection state version
> collection1 1
>    [junit4]   2> 2268330 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard2
>    [junit4]   2> 2268432 T6525 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2268432 T6518 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2268432 T6539 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2269328 T6540 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 2269329 T6540 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:collection1
>    [junit4]   2> 2269329 T6540 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 2269329 T6540 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 2269330 T6540 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 2269330 T6540 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/'
>    [junit4]   2> 2269331 T6540 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 2269331 T6540 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 2269331 T6540 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/lib/README'
> to classloader
>    [junit4]   2> 2269348 T6540 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 2269358 T6540 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 2269359 T6540 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 2269363 T6540 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 2269476 T6540 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 2269480 T6540 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 2269481 T6540 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 2269487 T6540 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 2269489 T6540 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 2269490 T6540 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 2269491 T6540
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 2269491 T6540
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 2269491 T6540 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 2269492 T6540
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 2269492 T6540
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 2269492 T6540 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 2269492 T6540 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 2269492 T6540 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/,
> dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1/
>    [junit4]   2> 2269492 T6540 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1aff83b8
>    [junit4]   2> 2269493 T6540 oasc.CachingDirectoryFactory.get return new
> directory for
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1
>    [junit4]   2> 2269493 T6540 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1/index/
>    [junit4]   2> 2269493 T6540 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 2269493 T6540 oasc.CachingDirectoryFactory.get return new
> directory for
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1/index
>    [junit4]   2> 2269494 T6540 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 2269494 T6540 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-336323af)),segFN=segments_1,generation=1}
>    [junit4]   2> 2269494 T6540 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 2269496 T6540 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 2269498 T6540 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 2269498 T6540 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 2269498 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2269498 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2269499 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2269499 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 2269499 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 2269499 T6540 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 2269500 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 2269500 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 2269500 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 2269500 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 2269501 T6540 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 2269502 T6540 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2269503 T6540 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2269503 T6540 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2269504 T6540 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2269505 T6540 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 2269505 T6540 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 2269506 T6540 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy:
> [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=25,
> maxMergedSegmentMB=14.1416015625, floorSegmentMB=1.9775390625,
> forceMergeDeletesPctAllowed=11.127099024208643, segmentsPerTier=45.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>    [junit4]   2> 2269506 T6540 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-336323af)),segFN=segments_1,generation=1}
>    [junit4]   2> 2269506 T6540 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 2269506 T6540 oass.SolrIndexSearcher.<init> Opening
> Searcher@10f6c7b7[collection1] main
>    [junit4]   2> 2269506 T6540 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 2269507 T6540 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 2269507 T6540 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 2269508 T6540
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 2269508 T6540 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 2269508 T6540 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 2269509 T6540
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38
> bytes from znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 2269509 T6540 oasr.ManagedResourceStorage.load Loaded
> LinkedHashMap at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 2269509 T6540 oasr.ManagedResource.reloadFromStorage
> Loaded initArgs {} for /rest/managed
>    [junit4]   2> 2269509 T6540 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 2269511 T6541 oasc.SolrCore.registerSearcher
> [collection1] Registered new searcher Searcher@10f6c7b7[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 2269512 T6540 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 2269513 T6544 oasc.ZkController.register Register replica
> - core:collection1 address:https://127.0.0.1:42284 collection:collection1
> shard:shard2
>    [junit4]   2> 2269513 T6489 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
>    [junit4]   2> 2269513 T6544 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard2/election
>    [junit4]   2> 2269513 T6489 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 2269516 T6544
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard2
>    [junit4]   2> 2269517 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2269517 T6544
> oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas
> found to continue.
>    [junit4]   2> 2269517 T6544
> oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader -
> try and sync
>    [junit4]   2> ASYNC  NEW_CORE C1658 name=collection1
> org.apache.solr.core.SolrCore@7683187f url=
> https://127.0.0.1:42284/collection1 node=127.0.0.1:42284_
> C1658_STATE=coll:collection1 core:collection1 props:{core=collection1,
> base_url=https://127.0.0.1:42284, node_name=127.0.0.1:42284_, state=down}
>    [junit4]   2> 2269518 T6544 C1658 P42284 oasc.SyncStrategy.sync Sync
> replicas to https://127.0.0.1:42284/collection1/
>    [junit4]   2> 2269518 T6544 C1658 P42284 oasc.SyncStrategy.syncReplicas
> Sync Success - now sync replicas to me
>    [junit4]   2> 2269518 T6544 C1658 P42284 oasc.SyncStrategy.syncToMe
> https://127.0.0.1:42284/collection1/ has no replicas
>    [junit4]   2> 2269518 T6544
> oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader:
> https://127.0.0.1:42284/collection1/ shard2
>    [junit4]   2> 2269518 T6544 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leaders/shard2
>    [junit4]   2> 2269521 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2269623 T6525 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2269624 T6539 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2269623 T6518 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2269672 T6544 oasc.ZkController.register We are
> https://127.0.0.1:42284/collection1/ and leader is
> https://127.0.0.1:42284/collection1/
>    [junit4]   2> 2269673 T6544 oasc.ZkController.register No LogReplay
> needed for core=collection1 baseURL=https://127.0.0.1:42284
>    [junit4]   2> 2269673 T6544 oasc.ZkController.checkRecovery I am the
> leader, no recovery necessary
>    [junit4]   2> 2269673 T6544 oasc.ZkController.publish publishing
> core=collection1 state=active collection=collection1
>    [junit4]   2> 2269673 T6544 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 2269674 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2269676 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "core_node_name":"core_node1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "base_url":"https://127.0.0.1:42284",
>    [junit4]   2>          "node_name":"127.0.0.1:42284_",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "state":"active",
>    [junit4]   2>          "shard":"shard2",
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "operation":"state"}
>    [junit4]   2> 2269778 T6525 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2269778 T6518 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2269779 T6539 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
>    [junit4]   2> 2269889 T6489
> oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
>    [junit4]   2> 2269890 T6489 oejs.Server.doStart jetty-8.1.10.v20130312
>    [junit4]   2> 2269893 T6489 oejus.SslContextFactory.doStart Enabled
> Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello,
> SSLv3, TLSv1, TLSv1.1, TLSv1.2]
>    [junit4]   2> 2269897 T6489 oejs.AbstractConnector.doStart Started
> SslSelectChannelConnector@127.0.0.1:42051
>    [junit4]   2> 2269899 T6489 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init()
>    [junit4]   2> 2269899 T6489 oasc.SolrResourceLoader.locateSolrHome JNDI
> not configured for solr (NoInitialContextEx)
>    [junit4]   2> 2269899 T6489 oasc.SolrResourceLoader.locateSolrHome
> using system property solr.solr.home:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004
>    [junit4]   2> 2269900 T6489 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/'
>    [junit4]   2> 2269918 T6489 oasc.ConfigSolr.fromFile Loading container
> configuration from
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/solr.xml
>    [junit4]   2> 2269928 T6489 oasc.CoreContainer.<init> New CoreContainer
> 603230535
>    [junit4]   2> 2269928 T6489 oasc.CoreContainer.load Loading cores into
> CoreContainer
> [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/]
>    [junit4]   2> 2269929 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting socketTimeout to: 90000
>    [junit4]   2> 2269929 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting urlScheme to:
>    [junit4]   2> 2269929 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting connTimeout to: 15000
>    [junit4]   2> 2269930 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maxConnectionsPerHost to: 20
>    [junit4]   2> 2269930 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting corePoolSize to: 0
>    [junit4]   2> 2269930 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maximumPoolSize to: 2147483647
>    [junit4]   2> 2269931 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting maxThreadIdleTime to: 5
>    [junit4]   2> 2269931 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting sizeOfQueue to: -1
>    [junit4]   2> 2269931 T6489 oashc.HttpShardHandlerFactory.getParameter
> Setting fairnessPolicy to: false
>    [junit4]   2> 2269933 T6489 oasu.UpdateShardHandler.<init> Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=340000&connTimeout=45000&retry=false
>    [junit4]   2> 2269934 T6489 oasl.LogWatcher.createWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>    [junit4]   2> 2269934 T6489 oasl.LogWatcher.newRegisteredLogWatcher
> Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>    [junit4]   2> 2269935 T6489 oasc.CoreContainer.load Host Name: 127.0.0.1
>    [junit4]   2> 2269935 T6489 oasc.ZkContainer.initZooKeeper Zookeeper
> client=127.0.0.1:57711/solr
>    [junit4]   2> 2269935 T6489 oasc.ZkController.checkChrootPath zkHost
> includes chroot
>    [junit4]   2> 2269936 T6489
> oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default
> ZkCredentialsProvider
>    [junit4]   2> 2269937 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2269937 T6555 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@5f4a4c14
> name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2269938 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2269938 T6489 oascc.SolrZkClient.createZkACLProvider
> Using default ZkACLProvider
>    [junit4]   2> 2269939 T6489 oascc.ConnectionManager.waitForConnected
> Waiting for client to connect to ZooKeeper
>    [junit4]   2> 2269940 T6557 oascc.ConnectionManager.process Watcher
> org.apache.solr.common.cloud.ConnectionManager@312c13f7
> name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event
> WatchedEvent state:SyncConnected type:None path:null path:null type:None
>    [junit4]   2> 2269940 T6489 oascc.ConnectionManager.waitForConnected
> Client is connected to ZooKeeper
>    [junit4]   2> 2269944 T6489
> oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster
> state from ZooKeeper...
>    [junit4]   2> 2270947 T6489 oasc.ZkController.createEphemeralLiveNode
> Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42051_
>    [junit4]   2> 2270948 T6489 oascc.SolrZkClient.makePath makePath:
> /live_nodes/127.0.0.1:42051_
>    [junit4]   2> 2270950 T6489 oasc.Overseer.close Overseer (id=null)
> closing
>    [junit4]   2> 2270953 T6559 oasc.ZkController.publish publishing
> core=collection1 state=down collection=collection1
>    [junit4]   2> 2270954 T6559 oasc.ZkController.publish numShards not
> found on descriptor - reading it from system property
>    [junit4]   2> 2270954 T6559 oasc.ZkController.waitForCoreNodeName look
> for our core node name
>    [junit4]   2> 2270954 T6518
> oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on
> path: /overseer/queue state: SyncConnected type NodeChildrenChanged
>    [junit4]   2> 2270956 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2
> message={
>    [junit4]   2>          "core":"collection1",
>    [junit4]   2>          "roles":null,
>    [junit4]   2>          "base_url":"https://127.0.0.1:42051",
>    [junit4]   2>          "node_name":"127.0.0.1:42051_",
>    [junit4]   2>          "numShards":"2",
>    [junit4]   2>          "state":"down",
>    [junit4]   2>          "shard":null,
>    [junit4]   2>          "collection":"collection1",
>    [junit4]   2>          "operation":"state"}
>    [junit4]   2> 2270956 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Collection already exists
> with numShards=2
>    [junit4]   2> 2270957 T6514
> oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard
> shard=shard1
>    [junit4]   2> 2271059 T6518 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 2271059 T6525 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 2271059 T6539 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 2271059 T6558 oascc.ZkStateReader$2.process A cluster
> state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
>    [junit4]   2> 2271955 T6559 oasc.ZkController.waitForShardId waiting to
> find shard id in clusterstate for collection1
>    [junit4]   2> 2271955 T6559 oasc.ZkController.createCollectionZkNode
> Check for collection zkNode:collection1
>    [junit4]   2> 2271955 T6559 oasc.ZkController.createCollectionZkNode
> Collection zkNode exists
>    [junit4]   2> 2271956 T6559 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 2271956 T6559 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 2271957 T6559 oasc.SolrResourceLoader.<init> new
> SolrResourceLoader for directory:
> '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/'
>    [junit4]   2> 2271957 T6559 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/lib/.svn/'
> to classloader
>    [junit4]   2> 2271957 T6559 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/lib/classes/'
> to classloader
>    [junit4]   2> 2271957 T6559 oasc.SolrResourceLoader.replaceClassLoader
> Adding
> 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/lib/README'
> to classloader
>    [junit4]   2> 2271976 T6559 oasc.SolrConfig.<init> Using Lucene
> MatchVersion: 6.0.0
>    [junit4]   2> 2271987 T6559 oasc.SolrConfig.<init> Loaded SolrConfig:
> solrconfig.xml
>    [junit4]   2> 2271988 T6559 oass.IndexSchema.readSchema Reading Solr
> Schema from schema.xml
>    [junit4]   2> 2271994 T6559 oass.IndexSchema.readSchema [collection1]
> Schema name=test
>    [junit4]   2> 2272122 T6559 oass.OpenExchangeRatesOrgProvider.init
> Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
>    [junit4]   2> 2272126 T6559 oass.IndexSchema.readSchema default search
> field in schema is text
>    [junit4]   2> 2272128 T6559 oass.IndexSchema.readSchema unique key
> field: id
>    [junit4]   2> 2272135 T6559 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 2272138 T6559 oass.FileExchangeRateProvider.reload
> Reloading exchange rates from file currency.xml
>    [junit4]   2> 2272139 T6559 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 2272140 T6559
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 2272140 T6559
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 2272141 T6559 oass.OpenExchangeRatesOrgProvider.reload
> Reloading exchange rates from open-exchange-rates.json
>    [junit4]   2> 2272141 T6559
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key
> IMPORTANT NOTE
>    [junit4]   2> 2272141 T6559
> oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected
> key, got STRING
>    [junit4]   2> 2272142 T6559 oasc.CoreContainer.create Creating SolrCore
> 'collection1' using configuration from collection collection1
>    [junit4]   2> 2272142 T6559 oasc.SolrCore.initDirectoryFactory
> org.apache.solr.core.MockDirectoryFactory
>    [junit4]   2> 2272142 T6559 oasc.SolrCore.<init> [collection1] Opening
> new SolrCore at
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/,
> dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2/
>    [junit4]   2> 2272142 T6559 oasc.JmxMonitoredMap.<init> JMX monitoring
> is enabled. Adding Solr mbeans to JMX Server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@1aff83b8
>    [junit4]   2> 2272143 T6559 oasc.CachingDirectoryFactory.get return new
> directory for
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2
>    [junit4]   2> 2272143 T6559 oasc.SolrCore.getNewIndexDir New index
> directory detected: old=null
> new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2/index/
>    [junit4]   2> 2272143 T6559 oasc.SolrCore.initIndex WARN [collection1]
> Solr index directory
> '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2/index'
> doesn't exist. Creating new index...
>    [junit4]   2> 2272143 T6559 oasc.CachingDirectoryFactory.get return new
> directory for
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2/index
>    [junit4]   2> 2272144 T6559 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy:
> [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19,
> maxMergeSize=9223372036854775807,
> maxMergeSizeForForcedMerge=9223372036854775807,
> calibrateSizeByDeletes=true, maxMergeDocs=2147483647,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
>    [junit4]   2> 2272144 T6559 oasc.SolrDeletionPolicy.onCommit
> SolrDeletionPolicy.onCommit: commits: num=1
>    [junit4]   2>
> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-15451404)),segFN=segments_1,generation=1}
>    [junit4]   2> 2272144 T6559 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 2272147 T6559 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "nodistrib"
>    [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "dedupe"
>    [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "dedupe"
>    [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "stored_sig"
>    [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
>    [junit4]   2> 2272149 T6559 oasup.UpdateRequestProcessorChain.init
> creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 2272149 T6559 oasup.UpdateRequestProcessorChain.init
> inserting DistributedUpdateProcessorFactory into
> updateRequestProcessorChain "distrib-dup-test-chain-implicit"
>    [junit4]   2> 2272149 T6559 oasc.SolrCore.loadUpdateProcessorChains no
> updateRequestProcessorChain defined as default, creating implicit default
>    [junit4]   2> 2272150 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /update/csv: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2272150 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2272150 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
>    [junit4]   2> 2272151 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created standard: solr.StandardRequestHandler
>    [junit4]   2> 2272151 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /get: solr.RealTimeGetHandler
>    [junit4]   2> 2272151 T6559 oasc.RequestHandlers.initHandlersFromConfig
> adding lazy requestHandler: solr.ReplicationHandler
>    [junit4]   2> 2272151 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /replication: solr.ReplicationHandler
>    [junit4]   2> 2272152 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /update: solr.UpdateRequestHandler
>    [junit4]   2> 2272152 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/: org.apache.solr.handler.admin.AdminHandlers
>    [junit4]   2> 2272153 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/file: solr.admin.ShowFileRequestHandler
>    [junit4]   2> 2272153 T6559 oasc.RequestHandlers.initHandlersFromConfig
> created /admin/fileedit: solr.admin.EditFileRequestHandler
>    [junit4]   2> 2272155 T6559 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2272156 T6559 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2272157 T6559 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2272157 T6559 oashl.XMLLoader.init
> xsltCacheLifetimeSeconds=60
>    [junit4]   2> 2272159 T6559 oasu.CommitTracker.<init> Hard AutoCommit:
> disabled
>    [junit4]   2> 2272159 T6559 oasu.CommitTracker.<init> Soft AutoCommit:
> disabled
>    [junit4]   2> 2272159 T6559 oasu.RandomMergePolicy.<init>
> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy:
> [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=25,
> maxMergedSegmentMB=14.1416015625, floorSegmentMB=1.9775390625,
> forceMergeDeletesPctAllowed=11.127099024208643, segmentsPerTier=45.0,
> maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
>    [junit4]   2> 2272160 T6559 oasc.SolrDeletionPolicy.onInit
> SolrDeletionPolicy.onInit: commits: num=1
>    [junit4]   2>
> commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-15451404)),segFN=segments_1,generation=1}
>    [junit4]   2> 2272160 T6559 oasc.SolrDeletionPolicy.updateCommits
> newest commit generation = 1
>    [junit4]   2> 2272160 T6559 oass.SolrIndexSearcher.<init> Opening
> Searcher@368b713f[collection1] main
>    [junit4]   2> 2272161 T6559 oascc.ZkStateReader.readConfigName Load
> collection config from:/collections/collection1
>    [junit4]   2> 2272162 T6559 oascc.ZkStateReader.readConfigName
> path=/collections/collection1 configName=conf1 specified config exists in
> ZooKeeper
>    [junit4]   2> 2272162 T6559 oasr.ManagedResourceStorage.newStorageIO
> Setting up ZooKeeper-based storage for the RestManager with znodeBase:
> /configs/conf1
>    [junit4]   2> 2272162 T6559
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured
> ZooKeeperStorageIO with znodeBase: /configs/conf1
>    [junit4]   2> 2272162 T6559 oasr.RestManager.init Initializing
> RestManager with initArgs: {}
>    [junit4]   2> 2272163 T6559 oasr.ManagedResourceStorage.load Reading
> _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 2272163 T6559
> oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38
> bytes from znode /configs/conf1/_rest_managed.json
>    [junit4]   2> 2272164 T6559 oasr.ManagedResourceStorage.load Loaded
> LinkedHashMap at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf1
>    [junit4]   2> 2272164 T6559 oasr.ManagedResource.reloadFromStorage
> Loaded initArgs {} for /rest/managed
>    [junit4]   2> 2272164 T6559 oasr.RestManager.init Initializing 0
> registered ManagedResources
>    [junit4]   2> 2272166 T6560 oasc.SolrCore.registerSearcher
> [collection1] Registered new searcher Searcher@368b713f[collection1]
> main{ExitableDirectoryReader(UninvertingDirectoryReader())}
>    [junit4]   2> 2272168 T6559 oasc.CoreContainer.registerCore registering
> core: collection1
>    [junit4]   2> 2272169 T6563 oasc.ZkController.register Register replica
> - core:collection1 address:https://127.0.0.1:42051 collection:collection1
> shard:shard1
>    [junit4]   2> 2272169 T6489 oass.SolrDispatchFilter.init
> user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
>    [junit4]   2> 2272169 T6489 oass.SolrDispatchFilter.init
> SolrDispatchFilter.init() done
>    [junit4]   2> 2272169 T6563 oascc.SolrZkClient.makePath makePath:
> /collections/collection1/leader_elect/shard1/election
>    [junit4]   2> 2272172 T6563
> oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process
> for shard shard1
>    [junit4]   2> 2272173 T6518
> oasc.DistributedQueue$LatchChildWatcher.process Lat
>
> [...truncated too long message...]
>
> ocsWithMultipleTerms(ExitableDirectoryReaderTest.java:88)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> 2526524 T7144 oas.SolrTestCaseJ4.setUp ###Starting
> testPrefixQuery
>    [junit4]   2> 2526525 T7144 C2233 oasc.SolrCore.execute [collection1]
> webapp=null path=null params={q=name:a*&indent=true&timeAllowed=1} hits=399
> status=0 QTime=0
>    [junit4]   2> 2526526 T7144 oas.SolrTestCaseJ4.tearDown ###Ending
> testPrefixQuery
>    [junit4]   2> NOTE: reproduce with: ant test
> -Dtestcase=ExitableDirectoryReaderTest -Dtests.method=testPrefixQuery
> -Dtests.seed=B805ADAE0F26DDD0 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=es_PA -Dtests.timezone=Asia/Macau -Dtests.file.encoding=UTF-8
>    [junit4] FAILURE 0.01s J1 | ExitableDirectoryReaderTest.testPrefixQuery
> <<<
>    [junit4]    > Throwable #1: java.lang.AssertionError:
>    [junit4]    >        at
> __randomizedtesting.SeedInfo.seed([B805ADAE0F26DDD0:B5299EC6BFB3955]:0)
>    [junit4]    >        at
> org.apache.solr.SolrTestCaseJ4.assertQEx(SolrTestCaseJ4.java:850)
>    [junit4]    >        at
> org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery(ExitableDirectoryReaderTest.java:61)
>    [junit4]    >        at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> 2526528 T7144 oas.SolrTestCaseJ4.deleteCore ###deleteCore
>    [junit4]   2> 2526528 T7144 oasc.CoreContainer.shutdown Shutting down
> CoreContainer instance=1777398481
>    [junit4]   2> 2526529 T7144 oasc.SolrCore.close [collection1]  CLOSING
> SolrCore org.apache.solr.core.SolrCore@579d0f01
>    [junit4]   2> 2526529 T7144 oasu.DirectUpdateHandler2.close closing
> DirectUpdateHandler2{commits=1,autocommits=0,soft
> autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1197,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
>    [junit4]   2> 2526529 T7144 oasu.SolrCoreState.decrefSolrCoreState
> Closing SolrCoreState
>    [junit4]   2> 2526529 T7144 oasu.DefaultSolrCoreState.closeIndexWriter
> SolrCoreState ref count has reached 0 - closing IndexWriter
>    [junit4]   2> 2526529 T7144 oasu.DefaultSolrCoreState.closeIndexWriter
> closing IndexWriter with IndexWriterCloser
>    [junit4]   2> 2526530 T7144 oasc.SolrCore.closeSearcher [collection1]
> Closing main searcher on request.
>    [junit4]   2> 2526530 T7144 oasc.CachingDirectoryFactory.close Closing
> MockDirectoryFactory - 2 directories currently being tracked
>    [junit4]   2> 2526530 T7144
> oasc.CachingDirectoryFactory.closeCacheValue looking to close
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001/index
> [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001/index;done=false>>]
>    [junit4]   2> 2526530 T7144 oasc.CachingDirectoryFactory.close Closing
> directory:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001/index
>    [junit4]   2> 2526531 T7144
> oasc.CachingDirectoryFactory.closeCacheValue looking to close
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001
> [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001;done=false>>]
>    [junit4]   2> 2526531 T7144 oasc.CachingDirectoryFactory.close Closing
> directory:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001
>    [junit4]   2> NOTE: leaving temporary files on disk at:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001
>    [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50),
> sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {},
> locale=es_PA, timezone=Asia/Macau
>    [junit4]   2> NOTE: Linux 3.13.0-36-generic amd64/Oracle Corporation
> 1.8.0_20 (64-bit)/cpus=8,threads=1,free=322112776,total=490733568
>    [junit4]   2> NOTE: All tests run in this JVM: [ZkSolrClientTest,
> URLClassifyProcessorTest, LoggingHandlerTest, ClusterStateTest,
> TestQueryTypes, CollectionsAPIAsyncDistributedZkTest,
> RAMDirectoryFactoryTest, TestImplicitCoreProperties, TestRealTimeGet,
> FullSolrCloudDistribCmdsTest, TestInitQParser, SOLR749Test,
> ShardRoutingCustomTest, SolrInfoMBeanTest, CopyFieldTest,
> TermVectorComponentTest, CoreMergeIndexesAdminHandlerTest,
> CustomCollectionTest, CoreAdminRequestStatusTest,
> DistributedQueryElevationComponentTest, TestRecovery,
> TestFileDictionaryLookup, SolrIndexConfigTest, TestCharFilters,
> SuggesterFSTTest, ShardSplitTest, FieldAnalysisRequestHandlerTest,
> DateFieldTest, BJQParserTest, HardAutoCommitTest, TestXIncludeConfig,
> TestQuerySenderNoQuery, DocExpirationUpdateProcessorFactoryTest,
> SynonymTokenizerTest, TestSolrQueryParser, TestRequestStatusCollectionAPI,
> TestNRTOpen, DistributedTermsComponentTest, TestManagedSchema,
> DateRangeFieldTest, DebugComponentTest, TestSolrXml, TestLFUCache,
> TestSchemaNameResource, DirectUpdateHandlerOptimizeTest, OutputWriterTest,
> SchemaVersionSpecificBehaviorTest, HttpPartitionTest, TestCollectionAPI,
> TestIndexingPerformance, CoreAdminCreateDiscoverTest,
> BasicDistributedZk2Test, DistanceFunctionTest, ChaosMonkeySafeLeaderTest,
> StatsComponentTest, QueryElevationComponentTest, TestExtendedDismaxParser,
> DocValuesTest, TestSortingResponseWriter, TestIBSimilarityFactory,
> SolrRequestParserTest, TestQueryUtils, LukeRequestHandlerTest,
> SliceStateTest, TestReversedWildcardFilterFactory,
> TestPostingsSolrHighlighter, TestNoOpRegenerator, DeleteShardTest,
> TestJmxMonitoredMap, StressHdfsTest, TestCursorMarkWithoutUniqueKey,
> SpatialFilterTest, CoreContainerCoreInitFailuresTest,
> TermVectorComponentDistributedTest,
> DistributedQueryComponentOptimizationTest, BlockDirectoryTest,
> TestDistributedGrouping, SolrXmlInZkTest, TestCloudManagedSchemaConcurrent,
> AnalyticsMergeStrategyTest, AnalysisAfterCoreReloadTest,
> ReplicationFactorTest, TestAddFieldRealTimeGet,
> TestOverriddenPrefixQueryForCustomFieldType, TestAtomicUpdateErrorCases,
> TestUniqueKeyFieldResource, TestExpandComponent, ConvertedLegacyTest,
> SimplePostToolTest, OpenCloseCoreStressTest,
> DistributedIntervalFacetingTest, TestElisionMultitermQuery, TestUtils,
> DeleteInactiveReplicaTest, TestReplicationHandlerBackup,
> TestManagedSynonymFilterFactory, TestZkChroot, TestCoreContainer,
> TestDistribDocBasedVersion, TestPerFieldSimilarity, SampleTest,
> NumericFieldsTest, AutoCommitTest, TestValueSourceCache,
> BasicDistributedZkTest, CollectionsAPIDistributedZkTest, BasicZkTest,
> TestRandomDVFaceting, TestFaceting, TestStressReorder,
> TestMultiCoreConfBootstrap, TestReload, SimpleFacetsTest, TestSolr4Spatial,
> SolrCoreTest, TestGroupingSearch, BadIndexSchemaTest, TestSort,
> TestFiltering, TestBadConfig, SoftAutoCommitTest, CoreAdminHandlerTest,
> WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestUpdate,
> DefaultValueUpdateProcessorTest, TestRemoteStreaming, TestWriterPerf,
> DirectSolrSpellCheckerTest, PrimitiveFieldTypeTest,
> FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest,
> MoreLikeThisHandlerTest, FastVectorHighlighterTest,
> RegexBoostProcessorTest, TestJmxIntegration,
> UpdateRequestProcessorFactoryTest, JSONWriterTest, QueryParsingTest,
> BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest,
> TestQuerySenderListener, TestSolrIndexConfig, TestMergePolicyConfig,
> MultiTermTest, TestBinaryField, TestSearchPerf, MinimalSchemaTest,
> TestConfig, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties,
> DirectSolrConnectionTest, TestLuceneMatchVersion, TestPhraseSuggestions,
> TestDFRSimilarityFactory, TestDefaultSimilarityFactory,
> LegacyHTMLStripCharFilterTest, ResourceLoaderTest, DateMathParserTest,
> PrimUtilsTest, SpellingQueryConverterTest, TestDocumentBuilder,
> UUIDFieldTest, CircularListTest, CursorPagingTest, TestTolerantSearch,
> AliasIntegrationTest, ConnectionManagerTest, DeleteReplicaTest,
> MigrateRouteKeyTest, RemoteQueryErrorTest, TestShortCircuitedRequests,
> VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test,
> HdfsChaosMonkeySafeLeaderTest, HdfsRecoveryZkTest,
> ExitableDirectoryReaderTest]
>    [junit4] Completed on J1 in 0.84s, 2 tests, 2 failures <<< FAILURES!
>
> [...truncated 209 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:524: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:472: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1359:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961:
> There were test failures: 432 suites, 1778 tests, 3 failures, 44 ignored
> (22 assumptions)
>
> Total time: 95 minutes 26 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 64bit/jdk1.8.0_20
> -XX:-UseCompressedOops -XX:+UseG1GC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>


-- 

Anshum Gupta
http://www.anshumgupta.net

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20) - Build # 11329 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11329/
Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseG1GC

3 tests failed.
FAILED:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.testDistribSearch

Error Message:
no exception matching expected: 400: Request took too long during query expansion. Terminating request.

Stack Trace:
java.lang.AssertionError: no exception matching expected: 400: Request took too long during query expansion. Terminating request.
	at __randomizedtesting.SeedInfo.seed([B805ADAE0F26DDD0:39E323B67879BDEC]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertFail(CloudExitableDirectoryReaderTest.java:101)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:81)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTest(CloudExitableDirectoryReaderTest.java:54)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:869)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.ExitableDirectoryReaderTest.testQueriesOnDocsWithMultipleTerms

Error Message:


Stack Trace:
java.lang.AssertionError: 
	at __randomizedtesting.SeedInfo.seed([B805ADAE0F26DDD0:D6C59E7F2353A6FF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.assertQEx(SolrTestCaseJ4.java:850)
	at org.apache.solr.core.ExitableDirectoryReaderTest.testQueriesOnDocsWithMultipleTerms(ExitableDirectoryReaderTest.java:88)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery

Error Message:


Stack Trace:
java.lang.AssertionError: 
	at __randomizedtesting.SeedInfo.seed([B805ADAE0F26DDD0:B5299EC6BFB3955]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.assertQEx(SolrTestCaseJ4.java:850)
	at org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery(ExitableDirectoryReaderTest.java:61)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11818 lines...]
   [junit4] Suite: org.apache.solr.cloud.CloudExitableDirectoryReaderTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001
   [junit4]   2> 2265251 T6489 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 2265251 T6489 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2265255 T6489 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2265255 T6489 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2265256 T6490 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2265356 T6489 oasc.ZkTestServer.run start zk server on port:57711
   [junit4]   2> 2265356 T6489 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2265357 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2265360 T6496 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3805495f name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2265360 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2265360 T6489 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2265360 T6489 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2265363 T6489 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2265366 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2265366 T6498 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fcbaf45 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2265367 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2265367 T6489 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2265367 T6489 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2265368 T6489 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2265370 T6489 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2265371 T6489 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2265372 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2265372 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2265373 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2265374 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2265375 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2265375 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2265377 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2265377 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2265378 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2265378 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2265379 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2265380 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2265381 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2265381 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2265382 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2265382 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2265383 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2265384 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2265385 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2265385 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2265386 T6489 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2265386 T6489 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2265389 T6489 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2265390 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2265391 T6500 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@346df4e9 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2265391 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2265391 T6489 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2265694 T6489 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2265698 T6489 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2265703 T6489 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58197
   [junit4]   2> 2265705 T6489 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2265705 T6489 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2265705 T6489 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002
   [junit4]   2> 2265706 T6489 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/'
   [junit4]   2> 2265722 T6489 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/solr.xml
   [junit4]   2> 2265733 T6489 oasc.CoreContainer.<init> New CoreContainer 1607842688
   [junit4]   2> 2265733 T6489 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/]
   [junit4]   2> 2265734 T6489 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2265734 T6489 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2265735 T6489 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2265735 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2265735 T6489 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2265736 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2265736 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2265736 T6489 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2265737 T6489 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2265738 T6489 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2265739 T6489 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2265740 T6489 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2265740 T6489 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2265741 T6489 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
   [junit4]   2> 2265741 T6489 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2265741 T6489 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2265742 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2265743 T6511 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ec8692c name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2265743 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2265743 T6489 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2265745 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2265746 T6513 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6884a990 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2265746 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2265747 T6489 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2265748 T6489 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2265750 T6489 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2265751 T6489 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2265752 T6489 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2265753 T6489 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2265754 T6489 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58197_
   [junit4]   2> 2265754 T6489 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58197_
   [junit4]   2> 2265755 T6489 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2265756 T6489 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2265757 T6489 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2265758 T6489 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58197_
   [junit4]   2> 2265758 T6489 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2265759 T6489 oasc.Overseer.start Overseer (id=92516182783492100-127.0.0.1:58197_-n_0000000000) starting
   [junit4]   2> 2265760 T6489 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2265764 T6489 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2265764 T6515 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2265764 T6489 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2265766 T6489 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2265766 T6489 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2265768 T6514 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2265770 T6517 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2265770 T6517 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2265771 T6517 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2265771 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2265772 T6514 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58197",
   [junit4]   2> 	  "node_name":"127.0.0.1:58197_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2265773 T6514 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2265773 T6514 oasc.Overseer$ClusterStateUpdater.createCollection state version control_collection 1
   [junit4]   2> 2265773 T6514 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2265774 T6518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2266771 T6517 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2266771 T6517 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2266772 T6517 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2266772 T6517 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2266773 T6517 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2266773 T6517 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/'
   [junit4]   2> 2266773 T6517 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2266774 T6517 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2266774 T6517 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2266791 T6517 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2266844 T6517 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2266845 T6517 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2266850 T6517 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2266951 T6517 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2266955 T6517 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2266956 T6517 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2266962 T6517 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2266964 T6517 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2266965 T6517 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2266966 T6517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2266966 T6517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2266966 T6517 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2266967 T6517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2266967 T6517 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2266967 T6517 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2266967 T6517 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2266967 T6517 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data/
   [junit4]   2> 2266968 T6517 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aff83b8
   [junit4]   2> 2266968 T6517 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data
   [junit4]   2> 2266968 T6517 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data/index/
   [junit4]   2> 2266968 T6517 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2266969 T6517 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/control/data/index
   [junit4]   2> 2266969 T6517 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2266969 T6517 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-56990cda)),segFN=segments_1,generation=1}
   [junit4]   2> 2266969 T6517 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2266971 T6517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2266972 T6517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2266973 T6517 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2266973 T6517 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2266973 T6517 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2266973 T6517 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2266973 T6517 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2266974 T6517 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2266975 T6517 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2266975 T6517 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2266975 T6517 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2266975 T6517 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2266995 T6517 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2266996 T6517 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2266996 T6517 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2266997 T6517 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2266998 T6517 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2266998 T6517 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2266999 T6517 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=14.1416015625, floorSegmentMB=1.9775390625, forceMergeDeletesPctAllowed=11.127099024208643, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2266999 T6517 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-56990cda)),segFN=segments_1,generation=1}
   [junit4]   2> 2266999 T6517 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2266999 T6517 oass.SolrIndexSearcher.<init> Opening Searcher@74a8b8af[collection1] main
   [junit4]   2> 2266999 T6517 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2267000 T6517 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2267000 T6517 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2267001 T6517 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2267001 T6517 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2267001 T6517 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2267001 T6517 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2267002 T6517 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2267002 T6517 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2267003 T6517 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2267003 T6517 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2267003 T6517 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2267005 T6519 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@74a8b8af[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2267006 T6517 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2267006 T6522 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:58197 collection:control_collection shard:shard1
   [junit4]   2> 2267007 T6489 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2267007 T6489 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2267008 T6522 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2267009 T6489 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2267010 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2267010 T6524 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bae41b3 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2267011 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2267011 T6489 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2267012 T6489 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2267012 T6522 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2267013 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2267014 T6522 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2267014 T6489 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2267014 T6522 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1657 name=collection1 org.apache.solr.core.SolrCore@2cf5fa37 url=https://127.0.0.1:58197/collection1 node=127.0.0.1:58197_ C1657_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=https://127.0.0.1:58197, node_name=127.0.0.1:58197_, state=down}
   [junit4]   2> 2267014 T6522 C1657 P58197 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:58197/collection1/
   [junit4]   2> 2267015 T6522 C1657 P58197 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2267015 T6522 C1657 P58197 oasc.SyncStrategy.syncToMe https://127.0.0.1:58197/collection1/ has no replicas
   [junit4]   2> 2267015 T6522 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:58197/collection1/ shard1
   [junit4]   2> 2267015 T6522 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2267021 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2267022 T6518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2267022 T6525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2267070 T6522 oasc.ZkController.register We are https://127.0.0.1:58197/collection1/ and leader is https://127.0.0.1:58197/collection1/
   [junit4]   2> 2267070 T6522 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:58197
   [junit4]   2> 2267070 T6522 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2267071 T6522 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2267071 T6522 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2267072 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2267074 T6514 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:58197",
   [junit4]   2> 	  "node_name":"127.0.0.1:58197_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2267176 T6525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2267176 T6518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2267278 T6489 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2267279 T6489 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2267282 T6489 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2267283 T6489 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42284
   [junit4]   2> 2267285 T6489 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2267286 T6489 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2267286 T6489 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003
   [junit4]   2> 2267286 T6489 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/'
   [junit4]   2> 2267298 T6489 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/solr.xml
   [junit4]   2> 2267307 T6489 oasc.CoreContainer.<init> New CoreContainer 1034069164
   [junit4]   2> 2267307 T6489 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/]
   [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2267308 T6489 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2267309 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2267309 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2267309 T6489 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2267309 T6489 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2267311 T6489 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2267311 T6489 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2267311 T6489 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2267312 T6489 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2267312 T6489 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
   [junit4]   2> 2267312 T6489 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2267313 T6489 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2267313 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2267314 T6536 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5698aded name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2267314 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2267315 T6489 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2267316 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2267316 T6538 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fbfe7d0 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2267316 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2267319 T6489 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2268322 T6489 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42284_
   [junit4]   2> 2268322 T6489 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42284_
   [junit4]   2> 2268325 T6489 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2268327 T6540 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2268327 T6540 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2268328 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2268328 T6540 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2268329 T6514 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42284",
   [junit4]   2> 	  "node_name":"127.0.0.1:42284_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2268330 T6514 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2268330 T6514 oasc.Overseer$ClusterStateUpdater.createCollection state version collection1 1
   [junit4]   2> 2268330 T6514 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2268432 T6525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2268432 T6518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2268432 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2269328 T6540 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2269329 T6540 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2269329 T6540 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2269329 T6540 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2269330 T6540 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2269330 T6540 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/'
   [junit4]   2> 2269331 T6540 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2269331 T6540 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2269331 T6540 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2269348 T6540 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2269358 T6540 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2269359 T6540 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2269363 T6540 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2269476 T6540 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2269480 T6540 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2269481 T6540 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2269487 T6540 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2269489 T6540 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2269490 T6540 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2269491 T6540 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2269491 T6540 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2269491 T6540 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2269492 T6540 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2269492 T6540 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2269492 T6540 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2269492 T6540 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2269492 T6540 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1/
   [junit4]   2> 2269492 T6540 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aff83b8
   [junit4]   2> 2269493 T6540 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1
   [junit4]   2> 2269493 T6540 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1/index/
   [junit4]   2> 2269493 T6540 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2269493 T6540 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty1/index
   [junit4]   2> 2269494 T6540 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2269494 T6540 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-336323af)),segFN=segments_1,generation=1}
   [junit4]   2> 2269494 T6540 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2269496 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2269497 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2269498 T6540 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2269498 T6540 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2269498 T6540 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2269498 T6540 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2269499 T6540 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2269499 T6540 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2269499 T6540 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2269499 T6540 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2269500 T6540 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2269500 T6540 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2269500 T6540 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2269500 T6540 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2269501 T6540 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2269502 T6540 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2269503 T6540 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2269503 T6540 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2269504 T6540 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2269505 T6540 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2269505 T6540 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2269506 T6540 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=14.1416015625, floorSegmentMB=1.9775390625, forceMergeDeletesPctAllowed=11.127099024208643, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2269506 T6540 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-336323af)),segFN=segments_1,generation=1}
   [junit4]   2> 2269506 T6540 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2269506 T6540 oass.SolrIndexSearcher.<init> Opening Searcher@10f6c7b7[collection1] main
   [junit4]   2> 2269506 T6540 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2269507 T6540 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2269507 T6540 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2269508 T6540 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2269508 T6540 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2269508 T6540 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2269509 T6540 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2269509 T6540 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2269509 T6540 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2269509 T6540 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2269511 T6541 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10f6c7b7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2269512 T6540 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2269513 T6544 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:42284 collection:collection1 shard:shard2
   [junit4]   2> 2269513 T6489 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2269513 T6544 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2269513 T6489 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2269516 T6544 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2269517 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2269517 T6544 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2269517 T6544 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1658 name=collection1 org.apache.solr.core.SolrCore@7683187f url=https://127.0.0.1:42284/collection1 node=127.0.0.1:42284_ C1658_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=https://127.0.0.1:42284, node_name=127.0.0.1:42284_, state=down}
   [junit4]   2> 2269518 T6544 C1658 P42284 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:42284/collection1/
   [junit4]   2> 2269518 T6544 C1658 P42284 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2269518 T6544 C1658 P42284 oasc.SyncStrategy.syncToMe https://127.0.0.1:42284/collection1/ has no replicas
   [junit4]   2> 2269518 T6544 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:42284/collection1/ shard2
   [junit4]   2> 2269518 T6544 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2269521 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2269623 T6525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2269624 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2269623 T6518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2269672 T6544 oasc.ZkController.register We are https://127.0.0.1:42284/collection1/ and leader is https://127.0.0.1:42284/collection1/
   [junit4]   2> 2269673 T6544 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:42284
   [junit4]   2> 2269673 T6544 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2269673 T6544 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2269673 T6544 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2269674 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2269676 T6514 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42284",
   [junit4]   2> 	  "node_name":"127.0.0.1:42284_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2269778 T6525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2269778 T6518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2269779 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2269889 T6489 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2269890 T6489 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2269893 T6489 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2269897 T6489 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42051
   [junit4]   2> 2269899 T6489 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2269899 T6489 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2269899 T6489 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004
   [junit4]   2> 2269900 T6489 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/'
   [junit4]   2> 2269918 T6489 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/solr.xml
   [junit4]   2> 2269928 T6489 oasc.CoreContainer.<init> New CoreContainer 603230535
   [junit4]   2> 2269928 T6489 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/]
   [junit4]   2> 2269929 T6489 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2269929 T6489 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2269929 T6489 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2269930 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2269930 T6489 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2269930 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2269931 T6489 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2269931 T6489 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2269931 T6489 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2269933 T6489 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2269934 T6489 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2269934 T6489 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2269935 T6489 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2269935 T6489 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57711/solr
   [junit4]   2> 2269935 T6489 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2269936 T6489 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 2269937 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2269937 T6555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f4a4c14 name:ZooKeeperConnection Watcher:127.0.0.1:57711 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2269938 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2269938 T6489 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 2269939 T6489 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2269940 T6557 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@312c13f7 name:ZooKeeperConnection Watcher:127.0.0.1:57711/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2269940 T6489 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2269944 T6489 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2270947 T6489 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42051_
   [junit4]   2> 2270948 T6489 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42051_
   [junit4]   2> 2270950 T6489 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 2270953 T6559 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2270954 T6559 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2270954 T6559 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2270954 T6518 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2270956 T6514 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"https://127.0.0.1:42051",
   [junit4]   2> 	  "node_name":"127.0.0.1:42051_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 2270956 T6514 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 2270957 T6514 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2271059 T6518 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2271059 T6525 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2271059 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2271059 T6558 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2271955 T6559 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2271955 T6559 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2271955 T6559 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2271956 T6559 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2271956 T6559 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2271957 T6559 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/'
   [junit4]   2> 2271957 T6559 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2271957 T6559 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2271957 T6559 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2271976 T6559 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 2271987 T6559 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2271988 T6559 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2271994 T6559 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2272122 T6559 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2272126 T6559 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2272128 T6559 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2272135 T6559 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2272138 T6559 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2272139 T6559 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2272140 T6559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2272140 T6559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2272141 T6559 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2272141 T6559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2272141 T6559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2272142 T6559 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2272142 T6559 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2272142 T6559 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2/
   [junit4]   2> 2272142 T6559 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aff83b8
   [junit4]   2> 2272143 T6559 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2
   [junit4]   2> 2272143 T6559 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2/index/
   [junit4]   2> 2272143 T6559 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2272143 T6559 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CloudExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/tempDir-001/jetty2/index
   [junit4]   2> 2272144 T6559 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=19, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 2272144 T6559 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-15451404)),segFN=segments_1,generation=1}
   [junit4]   2> 2272144 T6559 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2272147 T6559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2272148 T6559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2272149 T6559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2272149 T6559 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2272149 T6559 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2272150 T6559 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2272150 T6559 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2272150 T6559 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2272151 T6559 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2272151 T6559 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2272151 T6559 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2272151 T6559 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2272152 T6559 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2272152 T6559 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2272153 T6559 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2272153 T6559 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2272155 T6559 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2272156 T6559 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2272157 T6559 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2272157 T6559 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2272159 T6559 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2272159 T6559 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2272159 T6559 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=35, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=14.1416015625, floorSegmentMB=1.9775390625, forceMergeDeletesPctAllowed=11.127099024208643, segmentsPerTier=45.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2272160 T6559 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory(id=lucene-15451404)),segFN=segments_1,generation=1}
   [junit4]   2> 2272160 T6559 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2272160 T6559 oass.SolrIndexSearcher.<init> Opening Searcher@368b713f[collection1] main
   [junit4]   2> 2272161 T6559 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2272162 T6559 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2272162 T6559 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2272162 T6559 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2272162 T6559 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2272163 T6559 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2272163 T6559 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2272164 T6559 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2272164 T6559 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2272164 T6559 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2272166 T6560 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@368b713f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2272168 T6559 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2272169 T6563 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:42051 collection:collection1 shard:shard1
   [junit4]   2> 2272169 T6489 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 2272169 T6489 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2272169 T6563 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2272172 T6563 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2272173 T6518 oasc.DistributedQueue$LatchChildWatcher.process Lat

[...truncated too long message...]

ocsWithMultipleTerms(ExitableDirectoryReaderTest.java:88)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2526524 T7144 oas.SolrTestCaseJ4.setUp ###Starting testPrefixQuery
   [junit4]   2> 2526525 T7144 C2233 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=name:a*&indent=true&timeAllowed=1} hits=399 status=0 QTime=0 
   [junit4]   2> 2526526 T7144 oas.SolrTestCaseJ4.tearDown ###Ending testPrefixQuery
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ExitableDirectoryReaderTest -Dtests.method=testPrefixQuery -Dtests.seed=B805ADAE0F26DDD0 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PA -Dtests.timezone=Asia/Macau -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.01s J1 | ExitableDirectoryReaderTest.testPrefixQuery <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: 
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([B805ADAE0F26DDD0:B5299EC6BFB3955]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQEx(SolrTestCaseJ4.java:850)
   [junit4]    > 	at org.apache.solr.core.ExitableDirectoryReaderTest.testPrefixQuery(ExitableDirectoryReaderTest.java:61)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2526528 T7144 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 2526528 T7144 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1777398481
   [junit4]   2> 2526529 T7144 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@579d0f01
   [junit4]   2> 2526529 T7144 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1197,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 2526529 T7144 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 2526529 T7144 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 2526529 T7144 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 2526530 T7144 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 2526530 T7144 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 2526530 T7144 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 2526530 T7144 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001/index
   [junit4]   2> 2526531 T7144 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001;done=false>>]
   [junit4]   2> 2526531 T7144 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.ExitableDirectoryReaderTest-B805ADAE0F26DDD0-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_PA, timezone=Asia/Macau
   [junit4]   2> NOTE: Linux 3.13.0-36-generic amd64/Oracle Corporation 1.8.0_20 (64-bit)/cpus=8,threads=1,free=322112776,total=490733568
   [junit4]   2> NOTE: All tests run in this JVM: [ZkSolrClientTest, URLClassifyProcessorTest, LoggingHandlerTest, ClusterStateTest, TestQueryTypes, CollectionsAPIAsyncDistributedZkTest, RAMDirectoryFactoryTest, TestImplicitCoreProperties, TestRealTimeGet, FullSolrCloudDistribCmdsTest, TestInitQParser, SOLR749Test, ShardRoutingCustomTest, SolrInfoMBeanTest, CopyFieldTest, TermVectorComponentTest, CoreMergeIndexesAdminHandlerTest, CustomCollectionTest, CoreAdminRequestStatusTest, DistributedQueryElevationComponentTest, TestRecovery, TestFileDictionaryLookup, SolrIndexConfigTest, TestCharFilters, SuggesterFSTTest, ShardSplitTest, FieldAnalysisRequestHandlerTest, DateFieldTest, BJQParserTest, HardAutoCommitTest, TestXIncludeConfig, TestQuerySenderNoQuery, DocExpirationUpdateProcessorFactoryTest, SynonymTokenizerTest, TestSolrQueryParser, TestRequestStatusCollectionAPI, TestNRTOpen, DistributedTermsComponentTest, TestManagedSchema, DateRangeFieldTest, DebugComponentTest, TestSolrXml, TestLFUCache, TestSchemaNameResource, DirectUpdateHandlerOptimizeTest, OutputWriterTest, SchemaVersionSpecificBehaviorTest, HttpPartitionTest, TestCollectionAPI, TestIndexingPerformance, CoreAdminCreateDiscoverTest, BasicDistributedZk2Test, DistanceFunctionTest, ChaosMonkeySafeLeaderTest, StatsComponentTest, QueryElevationComponentTest, TestExtendedDismaxParser, DocValuesTest, TestSortingResponseWriter, TestIBSimilarityFactory, SolrRequestParserTest, TestQueryUtils, LukeRequestHandlerTest, SliceStateTest, TestReversedWildcardFilterFactory, TestPostingsSolrHighlighter, TestNoOpRegenerator, DeleteShardTest, TestJmxMonitoredMap, StressHdfsTest, TestCursorMarkWithoutUniqueKey, SpatialFilterTest, CoreContainerCoreInitFailuresTest, TermVectorComponentDistributedTest, DistributedQueryComponentOptimizationTest, BlockDirectoryTest, TestDistributedGrouping, SolrXmlInZkTest, TestCloudManagedSchemaConcurrent, AnalyticsMergeStrategyTest, AnalysisAfterCoreReloadTest, ReplicationFactorTest, TestAddFieldRealTimeGet, TestOverriddenPrefixQueryForCustomFieldType, TestAtomicUpdateErrorCases, TestUniqueKeyFieldResource, TestExpandComponent, ConvertedLegacyTest, SimplePostToolTest, OpenCloseCoreStressTest, DistributedIntervalFacetingTest, TestElisionMultitermQuery, TestUtils, DeleteInactiveReplicaTest, TestReplicationHandlerBackup, TestManagedSynonymFilterFactory, TestZkChroot, TestCoreContainer, TestDistribDocBasedVersion, TestPerFieldSimilarity, SampleTest, NumericFieldsTest, AutoCommitTest, TestValueSourceCache, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, BasicZkTest, TestRandomDVFaceting, TestFaceting, TestStressReorder, TestMultiCoreConfBootstrap, TestReload, SimpleFacetsTest, TestSolr4Spatial, SolrCoreTest, TestGroupingSearch, BadIndexSchemaTest, TestSort, TestFiltering, TestBadConfig, SoftAutoCommitTest, CoreAdminHandlerTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestUpdate, DefaultValueUpdateProcessorTest, TestRemoteStreaming, TestWriterPerf, DirectSolrSpellCheckerTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, MoreLikeThisHandlerTest, FastVectorHighlighterTest, RegexBoostProcessorTest, TestJmxIntegration, UpdateRequestProcessorFactoryTest, JSONWriterTest, QueryParsingTest, BinaryUpdateRequestHandlerTest, CSVRequestHandlerTest, TestQuerySenderListener, TestSolrIndexConfig, TestMergePolicyConfig, MultiTermTest, TestBinaryField, TestSearchPerf, MinimalSchemaTest, TestConfig, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, DirectSolrConnectionTest, TestLuceneMatchVersion, TestPhraseSuggestions, TestDFRSimilarityFactory, TestDefaultSimilarityFactory, LegacyHTMLStripCharFilterTest, ResourceLoaderTest, DateMathParserTest, PrimUtilsTest, SpellingQueryConverterTest, TestDocumentBuilder, UUIDFieldTest, CircularListTest, CursorPagingTest, TestTolerantSearch, AliasIntegrationTest, ConnectionManagerTest, DeleteReplicaTest, MigrateRouteKeyTest, RemoteQueryErrorTest, TestShortCircuitedRequests, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsChaosMonkeySafeLeaderTest, HdfsRecoveryZkTest, ExitableDirectoryReaderTest]
   [junit4] Completed on J1 in 0.84s, 2 tests, 2 failures <<< FAILURES!

[...truncated 209 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:524: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:472: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1359: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:961: There were test failures: 432 suites, 1778 tests, 3 failures, 44 ignored (22 assumptions)

Total time: 95 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any