You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/08/06 06:26:58 UTC

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1753 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1753/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([E68B1B39E68A9BE3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([E68B1B39E68A9BE3]:0)




Build Log:
[...truncated 11462 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/init-core-data-001
   [junit4]   2> 3737158 T8841 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 3737159 T8841 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 3737173 T8841 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 3737175 T8841 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3737177 T8842 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 3737277 T8841 oasc.ZkTestServer.run start zk server on port:51435
   [junit4]   2> 3737280 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3737292 T8848 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a98fde name:ZooKeeperConnection Watcher:127.0.0.1:51435 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3737296 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3737296 T8841 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 3737319 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3737321 T8850 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4b0c2e0a name:ZooKeeperConnection Watcher:127.0.0.1:51435/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3737322 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3737322 T8841 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 3737335 T8841 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 3737346 T8841 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 3737353 T8841 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 3737362 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3737364 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3737377 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 3737378 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 3737388 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3737389 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3737400 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3737402 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3737414 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3737416 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3737428 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3737430 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 3737447 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3737448 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3737458 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3737459 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3737470 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3737471 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3737481 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3737482 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3737491 T8841 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3737492 T8841 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3739647 T8841 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3739663 T8841 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51438
   [junit4]   2> 3739665 T8841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3739667 T8841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3739667 T8841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002
   [junit4]   2> 3739667 T8841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002/'
   [junit4]   2> 3739748 T8841 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002/solr.xml
   [junit4]   2> 3739845 T8841 oasc.CoreContainer.<init> New CoreContainer 963457925
   [junit4]   2> 3739847 T8841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002/]
   [junit4]   2> 3739848 T8841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3739848 T8841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3739848 T8841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3739850 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3739850 T8841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3739851 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3739851 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3739851 T8841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3739851 T8841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3739853 T8841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 3739854 T8841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3739855 T8841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3739855 T8841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3739855 T8841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51435/solr
   [junit4]   2> 3739856 T8841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3739857 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3739861 T8861 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30517e37 name:ZooKeeperConnection Watcher:127.0.0.1:51435 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3739862 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3739870 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3739874 T8863 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@67e17feb name:ZooKeeperConnection Watcher:127.0.0.1:51435/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3739875 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3739880 T8841 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 3739888 T8841 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 3739897 T8841 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 3739908 T8841 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 3739919 T8841 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 3739931 T8841 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 3739939 T8841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51438_
   [junit4]   2> 3739942 T8841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51438_
   [junit4]   2> 3739952 T8841 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 3739960 T8841 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 3739972 T8841 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51438_
   [junit4]   2> 3739973 T8841 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 3739985 T8841 oasc.Overseer.start Overseer (id=92228260110401539-127.0.0.1:51438_-n_0000000000) starting
   [junit4]   2> 3739994 T8841 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 3740034 T8865 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 3740035 T8841 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 3740045 T8841 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 3740051 T8841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3740064 T8864 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 3740415 T8866 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 3740416 T8866 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3740419 T8866 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3740421 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3740427 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3740429 T8864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51438",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51438_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3740429 T8864 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 3740429 T8864 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3740438 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3740440 T8867 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3741421 T8866 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3741422 T8866 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 3741424 T8866 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3741424 T8866 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 3741427 T8866 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3741429 T8866 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002/collection1/'
   [junit4]   2> 3741431 T8866 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 3741431 T8866 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 3741432 T8866 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 3741536 T8866 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 3741637 T8866 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3741641 T8866 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3741653 T8866 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3742154 T8866 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3742157 T8866 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3742160 T8866 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3742169 T8866 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3742242 T8866 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 3742243 T8866 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 3742244 T8866 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-002/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/control/data/
   [junit4]   2> 3742244 T8866 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@248ae6c5
   [junit4]   2> 3742245 T8866 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/control/data
   [junit4]   2> 3742247 T8866 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/control/data/index/
   [junit4]   2> 3742248 T8866 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 3742249 T8866 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/control/data/index
   [junit4]   2> 3742251 T8866 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.255859375, noCFSRatio=0.1405944386175203]
   [junit4]   2> 3742257 T8866 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 3742257 T8866 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3742267 T8866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3742268 T8866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3742269 T8866 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3742270 T8866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3742270 T8866 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3742271 T8866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3742272 T8866 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3742273 T8866 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3742273 T8866 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3742275 T8866 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3742275 T8866 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3742276 T8866 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3742278 T8866 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3742278 T8866 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3742280 T8866 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3742280 T8866 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3742281 T8866 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3742282 T8866 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3742283 T8866 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3742284 T8866 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 3742285 T8866 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 3742286 T8866 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 3742300 T8866 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3742303 T8866 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3742307 T8866 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3742310 T8866 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3742314 T8866 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3742319 T8866 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3742320 T8866 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3742320 T8866 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=43.3017578125, floorSegmentMB=1.7880859375, forceMergeDeletesPctAllowed=8.513086569820027, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3742323 T8866 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 3742323 T8866 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3742324 T8866 oass.SolrIndexSearcher.<init> Opening Searcher@12d6553d[collection1] main
   [junit4]   2> 3742324 T8866 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 3742328 T8866 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3742328 T8866 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3742331 T8866 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3742332 T8866 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 3742332 T8866 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3742335 T8866 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3742335 T8866 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3742337 T8866 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 3742348 T8866 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3742349 T8866 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3742349 T8866 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 3742355 T8868 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12d6553d[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 3742357 T8866 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3742362 T8871 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51438 collection:control_collection shard:shard1
   [junit4]   2> 3742364 T8841 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3742366 T8841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3742366 T8871 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 3742371 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3742376 T8873 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@28051867 name:ZooKeeperConnection Watcher:127.0.0.1:51435/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3742376 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3742382 T8841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3742391 T8871 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 3742391 T8841 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3742414 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3742416 T8871 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3742416 T8871 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1924 name=collection1 org.apache.solr.core.SolrCore@2e2fb2f7 url=http://127.0.0.1:51438/collection1 node=127.0.0.1:51438_ C1924_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:51438, core=collection1, node_name=127.0.0.1:51438_}
   [junit4]   2> 3742417 T8871 C1924 P51438 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51438/collection1/
   [junit4]   2> 3742417 T8871 C1924 P51438 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3742418 T8871 C1924 P51438 oasc.SyncStrategy.syncToMe http://127.0.0.1:51438/collection1/ has no replicas
   [junit4]   2> 3742418 T8871 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51438/collection1/ shard1
   [junit4]   2> 3742419 T8871 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 3742427 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3742442 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3742446 T8867 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3742452 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3742481 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3742489 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3742595 T8867 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3742596 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3742621 T8871 oasc.ZkController.register We are http://127.0.0.1:51438/collection1/ and leader is http://127.0.0.1:51438/collection1/
   [junit4]   2> 3742621 T8871 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51438
   [junit4]   2> 3742621 T8871 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3742622 T8871 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 3742622 T8871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3742628 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3742630 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3742631 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3742634 T8871 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3742639 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3742642 T8864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51438",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51438_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 3742650 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3742759 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3742759 T8867 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 3744817 T8841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 3744821 T8841 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3744835 T8841 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51442
   [junit4]   2> 3744839 T8841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3744840 T8841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3744840 T8841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003
   [junit4]   2> 3744841 T8841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003/'
   [junit4]   2> 3744907 T8841 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003/solr.xml
   [junit4]   2> 3744976 T8841 oasc.CoreContainer.<init> New CoreContainer 1954383060
   [junit4]   2> 3744977 T8841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003/]
   [junit4]   2> 3744979 T8841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3744979 T8841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3744980 T8841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3744980 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3744980 T8841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3744981 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3744981 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3744981 T8841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3744982 T8841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3744982 T8841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 3744983 T8841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3744983 T8841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3744983 T8841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3744984 T8841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51435/solr
   [junit4]   2> 3744984 T8841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3744986 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3744989 T8886 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c9bb60b name:ZooKeeperConnection Watcher:127.0.0.1:51435 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3744991 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3744999 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3745003 T8888 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5c495b2b name:ZooKeeperConnection Watcher:127.0.0.1:51435/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3745004 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3745038 T8841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3746051 T8841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51442_
   [junit4]   2> 3746055 T8841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51442_
   [junit4]   2> 3746094 T8890 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 3746095 T8890 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3746098 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3746099 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3746100 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3746099 T8890 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3746105 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3746108 T8864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51442",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51442_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3746108 T8864 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 3746109 T8864 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 3746118 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3746119 T8891 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3746119 T8889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3746119 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3747103 T8890 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3747103 T8890 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 3747106 T8890 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3747106 T8890 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3747109 T8890 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3747109 T8890 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003/collection1/'
   [junit4]   2> 3747111 T8890 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 3747111 T8890 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 3747112 T8890 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 3747226 T8890 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 3747287 T8890 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3747291 T8890 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3747305 T8890 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3747791 T8890 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3747796 T8890 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3747797 T8890 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3747806 T8890 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3747953 T8890 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 3747953 T8890 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 3747954 T8890 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-003/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty1/
   [junit4]   2> 3747954 T8890 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@248ae6c5
   [junit4]   2> 3747955 T8890 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty1
   [junit4]   2> 3747956 T8890 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty1/index/
   [junit4]   2> 3747957 T8890 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 3747964 T8890 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty1/index
   [junit4]   2> 3747965 T8890 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.255859375, noCFSRatio=0.1405944386175203]
   [junit4]   2> 3747971 T8890 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 3747972 T8890 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3747989 T8890 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3747990 T8890 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3747991 T8890 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3747991 T8890 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3747992 T8890 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3747992 T8890 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3747993 T8890 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3747995 T8890 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3747995 T8890 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3747997 T8890 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3747997 T8890 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3747998 T8890 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3747999 T8890 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3748000 T8890 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3748000 T8890 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3748001 T8890 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3748001 T8890 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3748002 T8890 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3748002 T8890 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3748003 T8890 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 3748005 T8890 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 3748006 T8890 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 3748020 T8890 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3748023 T8890 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3748028 T8890 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3748030 T8890 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3748033 T8890 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3748038 T8890 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3748038 T8890 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3748040 T8890 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=43.3017578125, floorSegmentMB=1.7880859375, forceMergeDeletesPctAllowed=8.513086569820027, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3748042 T8890 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 3748043 T8890 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3748043 T8890 oass.SolrIndexSearcher.<init> Opening Searcher@3a3adc09[collection1] main
   [junit4]   2> 3748043 T8890 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3748050 T8890 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3748051 T8890 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3748053 T8890 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3748053 T8890 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 3748053 T8890 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3748056 T8890 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3748057 T8890 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3748057 T8890 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 3748058 T8890 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 3748066 T8892 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a3adc09[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 3748071 T8890 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3748073 T8895 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51442 collection:collection1 shard:shard1
   [junit4]   2> 3748074 T8841 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3748074 T8841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3748075 T8895 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 3748090 T8895 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 3748100 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3748107 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3748113 T8895 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3748115 T8895 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1925 name=collection1 org.apache.solr.core.SolrCore@5bd23464 url=http://127.0.0.1:51442/collection1 node=127.0.0.1:51442_ C1925_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51442, core=collection1, node_name=127.0.0.1:51442_}
   [junit4]   2> 3748117 T8895 C1925 P51442 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51442/collection1/
   [junit4]   2> 3748117 T8895 C1925 P51442 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3748118 T8895 C1925 P51442 oasc.SyncStrategy.syncToMe http://127.0.0.1:51442/collection1/ has no replicas
   [junit4]   2> 3748118 T8895 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51442/collection1/ shard1
   [junit4]   2> 3748119 T8895 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 3748142 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3748147 T8891 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3748150 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3748159 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3748165 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3748168 T8889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3748274 T8891 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3748281 T8889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3748282 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3748295 T8895 oasc.ZkController.register We are http://127.0.0.1:51442/collection1/ and leader is http://127.0.0.1:51442/collection1/
   [junit4]   2> 3748295 T8895 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51442
   [junit4]   2> 3748296 T8895 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3748296 T8895 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 3748297 T8895 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3748303 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3748305 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3748307 T8895 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3748305 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3748316 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3748318 T8864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51442",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51442_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 3748326 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3748435 T8889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3748436 T8867 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3748435 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 3750617 T8841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 3750619 T8841 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3750627 T8841 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51445
   [junit4]   2> 3750635 T8841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3750636 T8841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3750638 T8841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004
   [junit4]   2> 3750639 T8841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004/'
   [junit4]   2> 3750757 T8841 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004/solr.xml
   [junit4]   2> 3750851 T8841 oasc.CoreContainer.<init> New CoreContainer 1932622690
   [junit4]   2> 3750852 T8841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004/]
   [junit4]   2> 3750854 T8841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3750854 T8841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3750855 T8841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3750855 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3750855 T8841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3750855 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3750856 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3750856 T8841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3750857 T8841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3750858 T8841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 3750859 T8841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3750859 T8841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3750859 T8841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3750860 T8841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51435/solr
   [junit4]   2> 3750860 T8841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3750863 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3750867 T8906 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@359b9a03 name:ZooKeeperConnection Watcher:127.0.0.1:51435 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3750868 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3750877 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3750883 T8908 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f3cb940 name:ZooKeeperConnection Watcher:127.0.0.1:51435/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3750884 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3750917 T8841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3751929 T8841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51445_
   [junit4]   2> 3751932 T8841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51445_
   [junit4]   2> 3751970 T8910 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 3751970 T8910 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3751973 T8910 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3751973 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3751973 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3751973 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3751981 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3751984 T8864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51445",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51445_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 3751984 T8864 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 3751984 T8864 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 3751996 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3751997 T8889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3751998 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3751998 T8875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3751998 T8909 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3752975 T8910 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3752976 T8910 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 3752977 T8910 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 3752978 T8910 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3752981 T8910 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3752982 T8910 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004/collection1/'
   [junit4]   2> 3752984 T8910 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 3752985 T8910 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 3752987 T8910 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 3753091 T8910 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 3753144 T8910 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3753148 T8910 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 3753160 T8910 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 3753659 T8910 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 3753663 T8910 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3753665 T8910 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3753676 T8910 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 3753767 T8910 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 3753768 T8910 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 3753768 T8910 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-004/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty2/
   [junit4]   2> 3753769 T8910 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@248ae6c5
   [junit4]   2> 3753770 T8910 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty2
   [junit4]   2> 3753771 T8910 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty2/index/
   [junit4]   2> 3753772 T8910 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 3753775 T8910 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty2/index
   [junit4]   2> 3753776 T8910 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=33, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=1.255859375, noCFSRatio=0.1405944386175203]
   [junit4]   2> 3753783 T8910 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 3753784 T8910 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3753797 T8910 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3753798 T8910 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3753798 T8910 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3753798 T8910 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3753799 T8910 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3753799 T8910 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3753801 T8910 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3753802 T8910 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3753802 T8910 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3753803 T8910 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3753804 T8910 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3753805 T8910 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3753806 T8910 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3753806 T8910 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3753807 T8910 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 3753807 T8910 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 3753808 T8910 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 3753808 T8910 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3753809 T8910 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3753810 T8910 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 3753812 T8910 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 3753813 T8910 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 3753831 T8910 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3753834 T8910 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3753836 T8910 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3753839 T8910 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3753843 T8910 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3753849 T8910 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3753850 T8910 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3753851 T8910 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=31, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=43.3017578125, floorSegmentMB=1.7880859375, forceMergeDeletesPctAllowed=8.513086569820027, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 3753854 T8910 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 3753855 T8910 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3753855 T8910 oass.SolrIndexSearcher.<init> Opening Searcher@28bd4cc9[collection1] main
   [junit4]   2> 3753855 T8910 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 3753861 T8910 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 3753862 T8910 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3753864 T8910 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3753865 T8910 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 3753865 T8910 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3753869 T8910 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3753870 T8910 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3753870 T8910 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 3753870 T8910 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 3753877 T8911 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@28bd4cc9[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 3753884 T8910 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3753888 T8914 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51445 collection:collection1 shard:shard2
   [junit4]   2> 3753889 T8841 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 3753889 T8841 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 3753891 T8914 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 3753906 T8914 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 3753913 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3753915 T8914 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 3753915 T8914 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C1926 name=collection1 org.apache.solr.core.SolrCore@49c59e77 url=http://127.0.0.1:51445/collection1 node=127.0.0.1:51445_ C1926_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51445, core=collection1, node_name=127.0.0.1:51445_}
   [junit4]   2> 3753916 T8914 C1926 P51445 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51445/collection1/
   [junit4]   2> 3753916 T8914 C1926 P51445 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 3753916 T8914 C1926 P51445 oasc.SyncStrategy.syncToMe http://127.0.0.1:51445/collection1/ has no replicas
   [junit4]   2> 3753917 T8914 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51445/collection1/ shard2
   [junit4]   2> 3753917 T8914 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 3753935 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3753938 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3753953 T8875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3753954 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3753953 T8889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3753962 T8909 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3753973 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3753983 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3754094 T8909 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3754095 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3754094 T8889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3754096 T8875 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3754139 T8914 oasc.ZkController.register We are http://127.0.0.1:51445/collection1/ and leader is http://127.0.0.1:51445/collection1/
   [junit4]   2> 3754140 T8914 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51445
   [junit4]   2> 3754140 T8914 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 3754140 T8914 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 3754141 T8914 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3754148 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3754149 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3754149 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3754154 T8914 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3754158 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 3754161 T8864 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51445",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51445_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 3754168 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3754280 T8909 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3754280 T8867 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3754280 T8874 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3754280 T8889 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 3756443 T8841 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 3756445 T8841 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 3756453 T8841 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51448
   [junit4]   2> 3756457 T8841 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 3756458 T8841 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3756459 T8841 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-005
   [junit4]   2> 3756459 T8841 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-005/'
   [junit4]   2> 3756542 T8841 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-005/solr.xml
   [junit4]   2> 3756639 T8841 oasc.CoreContainer.<init> New CoreContainer 804479115
   [junit4]   2> 3756639 T8841 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-E68B1B39E68A9BE3-001/tempDir-005/]
   [junit4]   2> 3756641 T8841 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 3756642 T8841 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 3756643 T8841 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 3756643 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3756643 T8841 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3756643 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3756644 T8841 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3756644 T8841 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3756644 T8841 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3756645 T8841 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 3756646 T8841 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3756646 T8841 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3756646 T8841 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 3756646 T8841 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51435/solr
   [junit4]   2> 3756647 T8841 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 3756649 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3756655 T8925 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@13475e45 name:ZooKeeperConnection Watcher:127.0.0.1:51435 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3756657 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3756667 T8841 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 3756672 T8927 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73297c7b name:ZooKeeperConnection Watcher:127.0.0.1:51435/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3756677 T8841 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 3756709 T8841 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 3757725 T8841 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51448_
   [junit4]   2> 3757728 T8841 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51448_
   [junit4]   2> 3757763 T8929 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 3757764 T8929 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 3757766 T8929 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 3757767 T8867 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3757767 T8875 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3757767 T8891 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 3757775 T8864 oascc.ZkStateReader.updateClusterState Updating cloud state

[...truncated too long message...]

d tasks = 0, completed tasks = 3089]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:201)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 10972559 T8863 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4093203b rejected from java.util.concurrent.ThreadPoolExecutor@41c32654[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3089]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:201)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 10972559 T8863 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@499f2d48 rejected from java.util.concurrent.ThreadPoolExecutor@41c32654[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3089]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:110)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:201)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=ko_KR, timezone=America/Thunder_Bay
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_65 (64-bit)/cpus=2,threads=100,free=279036248,total=470876160
   [junit4]   2> NOTE: All tests run in this JVM: [TestDefaultSimilarityFactory, BadIndexSchemaTest, TestNoOpRegenerator, TestSimpleQParserPlugin, TestMaxScoreQueryParser, TestZkChroot, TestManagedSchema, CollectionsAPIAsyncDistributedZkTest, TestPartialUpdateDeduplication, StatelessScriptUpdateProcessorFactoryTest, ClusterStateUpdateTest, TestManagedStopFilterFactory, PingRequestHandlerTest, TestFastWriter, HighlighterConfigTest, DistributedQueryComponentCustomSortTest, RequiredFieldsTest, CustomCollectionTest, TestValueSourceCache, TestCopyFieldCollectionResource, TestDistributedMissingSort, ScriptEngineTest, XmlUpdateRequestHandlerTest, MultiTermTest, TestIBSimilarityFactory, SOLR749Test, TestNRTOpen, TestQuerySenderNoQuery, TestCoreContainer, TestRecovery, BlockDirectoryTest, ExternalFileFieldSortTest, NumericFieldsTest, SolrInfoMBeanTest, HardAutoCommitTest, TestSolrQueryParser, ReturnFieldsTest, TestReload, TestFieldTypeResource, UpdateRequestProcessorFactoryTest, DeleteInactiveReplicaTest, TestDistributedGrouping, DocValuesMissingTest, TestLuceneMatchVersion, DistanceFunctionTest, AlternateDirectoryTest, IndexSchemaRuntimeFieldTest, TestHighFrequencyDictionaryFactory, TestJmxIntegration, TestShortCircuitedRequests, DirectSolrConnectionTest, TestStandardQParsers, DeleteReplicaTest, DistributedDebugComponentTest, ChaosMonkeyNothingIsSafeTest, TestElisionMultitermQuery, TestFaceting, CoreAdminCreateDiscoverTest, TestFastLRUCache, TestShardHandlerFactory, DateFieldTest, SearchHandlerTest, TestHashPartitioner, AutoCommitTest, TestCollapseQParserPlugin, PreAnalyzedUpdateProcessorTest, CoreContainerCoreInitFailuresTest, TestCollectionAPI, TestLMJelinekMercerSimilarityFactory, ShardRoutingCustomTest, RankQueryTest, DateRangeFieldTest, BasicZkTest, TestStressReorder, SystemInfoHandlerTest, CopyFieldTest, TestInfoStreamLogging, TriLevelCompositeIdRoutingTest, DebugComponentTest, DeleteShardTest, TestDocSet, TestSerializedLuceneMatchVersion, FieldAnalysisRequestHandlerTest, TestSolrDeletionPolicy2, TestPhraseSuggestions, JsonLoaderTest, TestNonDefinedSimilarityFactory, TestInitQParser, TestSolrJ, TestCursorMarkWithoutUniqueKey, RequestHandlersTest, TestUtils, ModifyConfFileTest, OpenCloseCoreStressTest, TestCodecSupport, IndexSchemaTest, TestAnalyzedSuggestions, CoreAdminHandlerTest, TestBinaryResponseWriter, RollingRestartTest, TestRandomMergePolicy, AddSchemaFieldsUpdateProcessorFactoryTest, TestFunctionQuery, EchoParamsTest, JSONWriterTest, SolrTestCaseJ4Test, TestLRUCache, SuggestComponentTest, HdfsLockFactoryTest, CachingDirectoryFactoryTest, QueryResultKeyTest, TestBM25SimilarityFactory, TestSweetSpotSimilarityFactory, TestSort, FastVectorHighlighterTest, DOMUtilTest, DocValuesTest, TestExtendedDismaxParser, TestJmxMonitoredMap, TestSearchPerf, QueryParsingTest, SolrCoreCheckLockOnStartupTest, DistributedSpellCheckComponentTest, AtomicUpdatesTest, SimplePostToolTest, SolrCmdDistributorTest, FieldMutatingUpdateProcessorTest, AliasIntegrationTest, TestCollationFieldDocValues, AssignTest, XsltUpdateRequestHandlerTest, TestComponentsName, TestLeaderElectionZkExpiry, SpatialFilterTest, SolrPluginUtilsTest, TestDynamicFieldCollectionResource, TestFreeTextSuggestions, TestFieldCollectionResource, FullSolrCloudDistribCmdsTest, TestRecoveryHdfs, TestFastOutputStream, TestSolrIndexConfig, AnalyticsQueryTest, WordBreakSolrSpellCheckerTest, PrimitiveFieldTypeTest, TestSolrQueryParserResource, TestQueryTypes, SyncSliceTest, SpellPossibilityIteratorTest, TestLFUCache, MultiThreadedOCPTest, TestLMDirichletSimilarityFactory, TestExpandComponent, TestNonNRTOpen, TestStressUserVersions, TestDistributedSearch, TestPHPSerializedResponseWriter, TestDocumentBuilder, RemoteQueryErrorTest, PreAnalyzedFieldTest, TestConfigSets, SoftAutoCommitTest, StressHdfsTest, TestIndexingPerformance, StandardRequestHandlerTest, TestSuggestSpellingConverter, EnumFieldTest, RAMDirectoryFactoryTest, TestReplicationHandlerBackup, TermsComponentTest, OverseerCollectionProcessorTest, HdfsCollectionsAPIDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, ZkControllerTest, SolrIndexConfigTest, InfoHandlerTest, SynonymTokenizerTest, CurrencyFieldXmlFileTest, TestRestManager, DistributedExpandComponentTest, FieldFacetExtrasTest, TestStressRecovery, FileBasedSpellCheckerTest, TestReloadAndDeleteDocs, TestSchemaVersionResource, MigrateRouteKeyTest, HttpPartitionTest, TestStressLucene, ShowFileRequestHandlerTest, TestSearcherReuse, AsyncMigrateRouteKeyTest, SpellingQueryConverterTest, TimeZoneUtilsTest, BJQParserTest, TestOmitPositions, SolrRequestParserTest, TestManagedSchemaFieldResource, HdfsUnloadDistributedZkTest, SolrXmlInZkTest, TestModifyConfFiles, MBeansHandlerTest, ConvertedLegacyTest, FunctionTest, HdfsBasicDistributedZkTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=E68B1B39E68A9BE3 -Dtests.slow=true -Dtests.locale=ko_KR -Dtests.timezone=America/Thunder_Bay -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E68B1B39E68A9BE3]:0)
   [junit4] Completed in 7235.47s, 1 test, 2 errors <<< FAILURES!

[...truncated 634 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1298: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:922: There were test failures: 422 suites, 1757 tests, 1 suite-level error, 1 error, 941 ignored (15 assumptions)

Total time: 235 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1755 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1755/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.schema.TestCloudSchemaless.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:53400/_/zl/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:53400/_/zl/collection1
	at __randomizedtesting.SeedInfo.seed([D9102A635FA5AA4:8C778CBE42A53A98]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:561)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
	at org.apache.solr.schema.TestCloudSchemaless.doTest(TestCloudSchemaless.java:140)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:911)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:449)
	... 46 more




Build Log:
[...truncated 12011 lines...]
   [junit4] Suite: org.apache.solr.schema.TestCloudSchemaless
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/init-core-data-001
   [junit4]   2> 6784500 T16873 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 6784501 T16873 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_/zl
   [junit4]   2> 6784514 T16873 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 6784515 T16873 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 6784518 T16874 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 6784618 T16873 oasc.ZkTestServer.run start zk server on port:53376
   [junit4]   2> 6784628 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6784638 T16880 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a7a097d name:ZooKeeperConnection Watcher:127.0.0.1:53376 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6784639 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6784639 T16873 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 6784652 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6784656 T16882 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@534de2f0 name:ZooKeeperConnection Watcher:127.0.0.1:53376/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6784656 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6784656 T16873 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 6784665 T16873 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 6784675 T16873 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 6784691 T16873 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 6784701 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-schemaless.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 6784702 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 6784719 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema-add-schema-fields-update-processor.xml to /configs/conf1/schema.xml
   [junit4]   2> 6784720 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 6784730 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 6784731 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 6784744 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 6784745 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 6784760 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 6784762 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 6784777 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 6784778 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 6784793 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 6784794 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 6784808 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 6784809 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 6784817 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 6784819 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 6784828 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 6784829 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 6784837 T16873 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 6784840 T16873 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 6784854 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6784856 T16884 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@372b626f name:ZooKeeperConnection Watcher:127.0.0.1:53376/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6784857 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6787440 T16873 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 6787453 T16873 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 6787461 T16873 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53380
   [junit4]   2> 6787467 T16873 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 6787468 T16873 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 6787469 T16873 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002
   [junit4]   2> 6787469 T16873 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/'
   [junit4]   2> 6787543 T16873 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/solr.xml
   [junit4]   2> 6787570 T16873 oasc.CoreContainer.<init> New CoreContainer 108815003
   [junit4]   2> 6787572 T16873 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/]
   [junit4]   2> 6787574 T16873 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 6787574 T16873 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 6787575 T16873 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 6787576 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 6787576 T16873 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 6787577 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 6787577 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 6787578 T16873 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 6787579 T16873 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 6787583 T16873 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 6787587 T16873 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 6787588 T16873 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 6787588 T16873 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 6787589 T16873 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53376/solr
   [junit4]   2> 6787590 T16873 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 6787595 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6787598 T16895 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@239f7b5f name:ZooKeeperConnection Watcher:127.0.0.1:53376 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6787599 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6787609 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6787614 T16897 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@63a02187 name:ZooKeeperConnection Watcher:127.0.0.1:53376/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6787616 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6787620 T16873 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 6787632 T16873 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 6787647 T16873 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 6787660 T16873 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 6787676 T16873 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 6787712 T16873 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 6787719 T16873 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53380__%2Fzl
   [junit4]   2> 6787722 T16873 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53380__%2Fzl
   [junit4]   2> 6787737 T16873 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 6787753 T16873 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 6787778 T16873 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:53380__%2Fzl
   [junit4]   2> 6787780 T16873 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 6787791 T16873 oasc.Overseer.start Overseer (id=92234513360683012-127.0.0.1:53380__%2Fzl-n_0000000000) starting
   [junit4]   2> 6787801 T16873 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 6787869 T16898 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 6787871 T16873 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 6787869 T16899 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 6787883 T16873 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 6787890 T16873 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 6787912 T16900 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 6787912 T16900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 6787915 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6787915 T16900 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 6787922 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6787925 T16898 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53380/_/zl",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53380__%2Fzl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 6787926 T16898 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 6787926 T16898 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 6787937 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6787938 T16901 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 6788919 T16900 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 6788920 T16900 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 6788922 T16900 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 6788922 T16900 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 6788926 T16900 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 6788927 T16900 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/'
   [junit4]   2> 6788930 T16900 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 6788932 T16900 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 6788933 T16900 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 6789003 T16900 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 6789014 T16900 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 6789018 T16900 oass.ManagedIndexSchemaFactory.create The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 6789022 T16900 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 6789027 T16900 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 6789067 T16900 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 6789072 T16900 oass.ManagedIndexSchema.persistManagedSchemaToZooKeeper Created and persisted managed schema znode at /configs/conf1/managed-schema
   [junit4]   2> 6789078 T16900 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml.bak
   [junit4]   2> 6789097 T16900 oass.ManagedIndexSchemaFactory.zkUgradeToManagedSchema After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/conf1/schema.xml to /configs/conf1/schema.xml.bak
   [junit4]   2> 6789098 T16900 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 6789098 T16900 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 6789099 T16900 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/data/
   [junit4]   2> 6789099 T16900 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 6789101 T16900 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/data
   [junit4]   2> 6789102 T16900 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/data/index/
   [junit4]   2> 6789103 T16900 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 6789104 T16900 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/data/index
   [junit4]   2> 6789105 T16900 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=39.5703125, floorSegmentMB=1.9560546875, forceMergeDeletesPctAllowed=8.838015237842777, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2737781094873187
   [junit4]   2> 6789110 T16900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@618e776; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 6789111 T16900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6789113 T16900 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 6789115 T16900 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 6789116 T16900 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6789116 T16900 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6789116 T16900 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6789117 T16900 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6789117 T16900 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 6789118 T16900 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 6789118 T16900 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 6789119 T16900 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6789119 T16900 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 6789119 T16900 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 6789120 T16900 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 6789128 T16900 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6789130 T16900 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6789132 T16900 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6789133 T16900 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6789135 T16900 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6789138 T16900 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 6789138 T16900 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 6789139 T16900 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=65.6923828125, floorSegmentMB=1.1259765625, forceMergeDeletesPctAllowed=24.303715619738007, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13084911301462637
   [junit4]   2> 6789141 T16900 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-002/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@618e776; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 6789142 T16900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6789142 T16900 oass.SolrIndexSearcher.<init> Opening Searcher@8b698ea[collection1] main
   [junit4]   2> 6789143 T16900 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 6789146 T16900 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 6789147 T16900 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 6789149 T16900 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 6789150 T16900 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 6789150 T16900 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 6789152 T16900 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 6789153 T16900 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 6789155 T16900 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 6789163 T16900 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 6789164 T16900 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 6789165 T16900 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 6789165 T16900 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 6789169 T16900 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 6789169 T16902 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8b698ea[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 6789172 T16905 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53380/_/zl collection:control_collection shard:shard1
   [junit4]   2> 6789172 T16873 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 6789173 T16873 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 6789175 T16905 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 6789181 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6789185 T16907 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ab8049f name:ZooKeeperConnection Watcher:127.0.0.1:53376/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6789186 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6789189 T16873 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 6789193 T16905 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 6789195 T16873 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 6789197 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6789201 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6789206 T16905 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 6789207 T16905 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 6789208 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> ASYNC  NEW_CORE C13181 name=collection1 org.apache.solr.core.SolrCore@54c0e475 url=https://127.0.0.1:53380/_/zl/collection1 node=127.0.0.1:53380__%2Fzl C13181_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:53380/_/zl, core=collection1, node_name=127.0.0.1:53380__%2Fzl}
   [junit4]   2> 6789211 T16905 C13181 P53380 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:53380/_/zl/collection1/
   [junit4]   2> 6789212 T16905 C13181 P53380 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 6789212 T16905 C13181 P53380 oasc.SyncStrategy.syncToMe https://127.0.0.1:53380/_/zl/collection1/ has no replicas
   [junit4]   2> 6789213 T16905 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:53380/_/zl/collection1/ shard1
   [junit4]   2> 6789213 T16905 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 6789233 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6789261 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6789367 T16908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 6789371 T16901 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 6789403 T16905 oasc.ZkController.register We are https://127.0.0.1:53380/_/zl/collection1/ and leader is https://127.0.0.1:53380/_/zl/collection1/
   [junit4]   2> 6789403 T16905 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53380/_/zl
   [junit4]   2> 6789404 T16905 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 6789404 T16905 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 6789404 T16905 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 6789408 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6789410 T16909 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6789411 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6789412 T16905 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6789417 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6789421 T16898 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53380/_/zl",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53380__%2Fzl",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 6789430 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6789543 T16908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 6789545 T16901 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 6791635 T16873 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 6791637 T16873 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 6791651 T16873 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 6791654 T16873 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53384
   [junit4]   2> 6791659 T16873 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 6791659 T16873 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 6791660 T16873 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003
   [junit4]   2> 6791660 T16873 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/'
   [junit4]   2> 6791713 T16873 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/solr.xml
   [junit4]   2> 6791728 T16873 oasc.CoreContainer.<init> New CoreContainer 123091986
   [junit4]   2> 6791729 T16873 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/]
   [junit4]   2> 6791729 T16873 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 6791730 T16873 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 6791730 T16873 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 6791730 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 6791731 T16873 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 6791731 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 6791732 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 6791732 T16873 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 6791733 T16873 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 6791735 T16873 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 6791739 T16873 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 6791740 T16873 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 6791741 T16873 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 6791742 T16873 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53376/solr
   [junit4]   2> 6791742 T16873 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 6791744 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6791748 T16920 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a2f6c20 name:ZooKeeperConnection Watcher:127.0.0.1:53376 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6791749 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6791756 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6791758 T16922 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@116d8ecb name:ZooKeeperConnection Watcher:127.0.0.1:53376/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6791759 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6791792 T16873 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 6792806 T16873 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53384__%2Fzl
   [junit4]   2> 6792809 T16873 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53384__%2Fzl
   [junit4]   2> 6792837 T16924 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 6792837 T16924 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 6792841 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6792841 T16909 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6792841 T16924 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 6792844 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6792848 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6792850 T16898 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53384/_/zl",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53384__%2Fzl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 6792850 T16898 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 6792851 T16898 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 6792856 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6792859 T16923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6792860 T16908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6792861 T16925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6793846 T16924 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 6793846 T16924 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 6793848 T16924 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 6793849 T16924 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 6793853 T16924 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 6793854 T16924 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/'
   [junit4]   2> 6793856 T16924 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 6793858 T16924 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 6793859 T16924 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 6793927 T16924 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 6793938 T16924 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 6793947 T16924 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 6793953 T16924 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 6793978 T16924 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 6793979 T16924 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 6793980 T16924 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 6793982 T16924 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/data/
   [junit4]   2> 6793982 T16924 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 6793983 T16924 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/data
   [junit4]   2> 6793985 T16924 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/data/index/
   [junit4]   2> 6793991 T16924 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 6793995 T16924 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/data/index
   [junit4]   2> 6793996 T16924 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=39.5703125, floorSegmentMB=1.9560546875, forceMergeDeletesPctAllowed=8.838015237842777, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2737781094873187
   [junit4]   2> 6794010 T16924 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@347cb538; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 6794013 T16924 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6794018 T16924 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 6794023 T16924 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 6794023 T16924 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6794024 T16924 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6794027 T16924 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6794028 T16924 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6794028 T16924 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 6794029 T16924 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 6794029 T16924 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 6794029 T16924 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6794030 T16924 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 6794030 T16924 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 6794031 T16924 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 6794059 T16924 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6794069 T16924 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6794071 T16924 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6794074 T16924 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6794076 T16924 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6794080 T16924 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 6794081 T16924 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 6794082 T16924 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=65.6923828125, floorSegmentMB=1.1259765625, forceMergeDeletesPctAllowed=24.303715619738007, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13084911301462637
   [junit4]   2> 6794086 T16924 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-003/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@347cb538; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 6794087 T16924 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6794089 T16924 oass.SolrIndexSearcher.<init> Opening Searcher@450bc28b[collection1] main
   [junit4]   2> 6794090 T16924 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 6794099 T16924 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 6794099 T16924 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 6794102 T16924 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 6794102 T16924 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 6794103 T16924 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 6794109 T16924 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 6794109 T16924 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 6794110 T16924 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 6794110 T16924 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 6794110 T16924 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 6794117 T16924 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 6794117 T16926 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@450bc28b[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 6794123 T16929 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53384/_/zl collection:collection1 shard:shard2
   [junit4]   2> 6794123 T16873 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 6794124 T16873 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 6794126 T16929 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 6794141 T16929 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 6794180 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6794185 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6794203 T16929 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 6794204 T16929 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C13182 name=collection1 org.apache.solr.core.SolrCore@27de221c url=https://127.0.0.1:53384/_/zl/collection1 node=127.0.0.1:53384__%2Fzl C13182_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:53384/_/zl, core=collection1, node_name=127.0.0.1:53384__%2Fzl}
   [junit4]   2> 6794205 T16929 C13182 P53384 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:53384/_/zl/collection1/
   [junit4]   2> 6794205 T16929 C13182 P53384 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 6794205 T16929 C13182 P53384 oasc.SyncStrategy.syncToMe https://127.0.0.1:53384/_/zl/collection1/ has no replicas
   [junit4]   2> 6794205 T16929 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:53384/_/zl/collection1/ shard2
   [junit4]   2> 6794206 T16929 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 6794227 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6794240 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6794252 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6794359 T16908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6794362 T16923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6794364 T16925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6794400 T16929 oasc.ZkController.register We are https://127.0.0.1:53384/_/zl/collection1/ and leader is https://127.0.0.1:53384/_/zl/collection1/
   [junit4]   2> 6794402 T16929 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53384/_/zl
   [junit4]   2> 6794402 T16929 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 6794403 T16929 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 6794403 T16929 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 6794406 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6794407 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6794406 T16909 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6794411 T16929 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6794414 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6794416 T16898 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53384/_/zl",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53384__%2Fzl",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 6794425 T16909 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6794538 T16908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6794539 T16923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6794540 T16909 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 6796560 T16873 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 6796561 T16873 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 6796572 T16873 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 6796575 T16873 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53387
   [junit4]   2> 6796579 T16873 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 6796579 T16873 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 6796579 T16873 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004
   [junit4]   2> 6796580 T16873 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/'
   [junit4]   2> 6796643 T16873 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/solr.xml
   [junit4]   2> 6796655 T16873 oasc.CoreContainer.<init> New CoreContainer 1852773864
   [junit4]   2> 6796656 T16873 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/]
   [junit4]   2> 6796657 T16873 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 6796657 T16873 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 6796658 T16873 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 6796658 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 6796658 T16873 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 6796659 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 6796659 T16873 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 6796659 T16873 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 6796660 T16873 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 6796662 T16873 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 6796664 T16873 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 6796664 T16873 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 6796665 T16873 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 6796665 T16873 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53376/solr
   [junit4]   2> 6796665 T16873 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 6796666 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6796669 T16940 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@124e895c name:ZooKeeperConnection Watcher:127.0.0.1:53376 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6796670 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6796674 T16873 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 6796677 T16942 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@572d5f31 name:ZooKeeperConnection Watcher:127.0.0.1:53376/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 6796677 T16873 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 6796695 T16873 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 6797706 T16873 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53387__%2Fzl
   [junit4]   2> 6797708 T16873 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53387__%2Fzl
   [junit4]   2> 6797746 T16944 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 6797746 T16944 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 6797748 T16909 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6797748 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6797748 T16944 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 6797748 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6797754 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6797756 T16898 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53387/_/zl",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53387__%2Fzl",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 6797756 T16898 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 6797756 T16898 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 6797762 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6797764 T16925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6797764 T16943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6797764 T16908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6797765 T16923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6798751 T16944 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 6798751 T16944 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 6798753 T16944 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 6798753 T16944 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 6798757 T16944 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 6798757 T16944 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/'
   [junit4]   2> 6798759 T16944 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 6798760 T16944 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 6798760 T16944 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 6798810 T16944 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 6798820 T16944 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 6798824 T16944 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 6798827 T16944 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 6798841 T16944 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 6798843 T16944 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 6798844 T16944 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 6798847 T16944 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/data/
   [junit4]   2> 6798847 T16944 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 6798850 T16944 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/data
   [junit4]   2> 6798852 T16944 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/data/index/
   [junit4]   2> 6798854 T16944 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 6798856 T16944 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/data/index
   [junit4]   2> 6798857 T16944 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=39.5703125, floorSegmentMB=1.9560546875, forceMergeDeletesPctAllowed=8.838015237842777, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2737781094873187
   [junit4]   2> 6798866 T16944 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7493e4ac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 6798866 T16944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6798871 T16944 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 6798875 T16944 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 6798876 T16944 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6798877 T16944 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6798878 T16944 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6798878 T16944 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6798880 T16944 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 6798880 T16944 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 6798881 T16944 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 6798882 T16944 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6798882 T16944 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 6798882 T16944 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 6798883 T16944 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 6798895 T16944 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6798899 T16944 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6798901 T16944 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6798906 T16944 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6798908 T16944 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6798911 T16944 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 6798912 T16944 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 6798912 T16944 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=28, maxMergeAtOnceExplicit=22, maxMergedSegmentMB=65.6923828125, floorSegmentMB=1.1259765625, forceMergeDeletesPctAllowed=24.303715619738007, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.13084911301462637
   [junit4]   2> 6798914 T16944 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-004/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7493e4ac; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 6798915 T16944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6798915 T16944 oass.SolrIndexSearcher.<init> Opening Searcher@21b2b505[collection1] main
   [junit4]   2> 6798916 T16944 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 6798920 T16944 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 6798920 T16944 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 6798922 T16944 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 6798922 T16944 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 6798923 T16944 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 6798926 T16944 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 6798926 T16944 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 6798927 T16944 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 6798927 T16944 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 6798927 T16944 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 6798931 T16944 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 6798931 T16945 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21b2b505[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 6798934 T16948 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:53387/_/zl collection:collection1 shard:shard3
   [junit4]   2> 6798937 T16873 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 6798938 T16873 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 6798939 T16948 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 6798955 T16948 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 6798964 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6798968 T16948 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 6798970 T16948 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C13183 name=collection1 org.apache.solr.core.SolrCore@7d661889 url=https://127.0.0.1:53387/_/zl/collection1 node=127.0.0.1:53387__%2Fzl C13183_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:53387/_/zl, core=collection1, node_name=127.0.0.1:53387__%2Fzl}
   [junit4]   2> 6798971 T16948 C13183 P53387 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:53387/_/zl/collection1/
   [junit4]   2> 6798972 T16948 C13183 P53387 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 6798972 T16948 C13183 P53387 oasc.SyncStrategy.syncToMe https://127.0.0.1:53387/_/zl/collection1/ has no replicas
   [junit4]   2> 6798972 T16948 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:53387/_/zl/collection1/ shard3
   [junit4]   2> 6798973 T16948 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 6798981 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6798994 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6799008 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6799117 T16943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6799118 T16925 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6799118 T16923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6799117 T16908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6799155 T16948 oasc.ZkController.register We are https://127.0.0.1:53387/_/zl/collection1/ and leader is https://127.0.0.1:53387/_/zl/collection1/
   [junit4]   2> 6799157 T16948 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:53387/_/zl
   [junit4]   2> 6799157 T16948 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 6799158 T16948 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 6799159 T16948 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 6799164 T16925 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6799165 T16909 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6799166 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6799170 T16948 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6799177 T16898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 6799179 T16898 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:53387/_/zl",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:53387__%2Fzl",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 6799191 T16901 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 6799299 T16923 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6799300 T16901 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6799303 T16943 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6799303 T16908 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 6801428 T16873 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 6801429 T16873 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 6801441 T16873 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 6801456 T16873 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:53390
   [junit4]   2> 6801460 T16873 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 6801468 T16873 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 6801470 T16873 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-005
   [junit4]   2> 6801470 T16873 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-005/'
   [junit4]   2> 6801524 T16873 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-005/solr.xml
   [junit4]   2> 6801542 T16873 oasc.CoreContainer.<init> New CoreContainer 2061571902
   [junit4]   2> 6801544 T16873 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-D9102A635FA5AA4-001/tempDir-005/]
   [junit4]   2> 6801545 T16873 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 6801545 T16

[...truncated too long message...]

riddenPrefixQueryForCustomFieldType, TestSolrQueryParserDefaultOperatorResource, TestFastOutputStream, TestMaxScoreQueryParser, TestDistributedSearch, TestNonNRTOpen, LeaderElectionTest, CursorPagingTest, TestCustomSort, TestDefaultSearchFieldResource, DisMaxRequestHandlerTest, DirectSolrConnectionTest, SimplePostToolTest, TestSerializedLuceneMatchVersion, TestPartialUpdateDeduplication, SolrIndexConfigTest, SolrCmdDistributorTest, DirectSolrSpellCheckerTest, CoreContainerCoreInitFailuresTest, FieldAnalysisRequestHandlerTest, TestFreeTextSuggestions, TestConfigSets, TestFiltering, TermVectorComponentTest, TestReloadAndDeleteDocs, TestFoldingMultitermQuery, RollingRestartTest, CachingDirectoryFactoryTest, TestStressReorder, TestMiniSolrCloudCluster, TestQueryTypes, ZkNodePropsTest, TestRecovery, PingRequestHandlerTest, TestLFUCache, CoreMergeIndexesAdminHandlerTest, CollectionsAPIDistributedZkTest, BinaryUpdateRequestHandlerTest, TestJoin, TestSolrXmlPersistence, TestQueryUtils, TestRestManager, CopyFieldTest, CursorMarkTest, RecoveryZkTest, TestStandardQParsers, SliceStateTest, TestCharFilters, BlockDirectoryTest, ShardSplitTest, HttpPartitionTest, PluginInfoTest, TestNonDefinedSimilarityFactory, SearchHandlerTest, FastVectorHighlighterTest, TestExtendedDismaxParser, SuggesterFSTTest, TestDefaultSimilarityFactory, SignatureUpdateProcessorFactoryTest, HdfsUnloadDistributedZkTest, TestLMDirichletSimilarityFactory, StatelessScriptUpdateProcessorFactoryTest, LeaderElectionIntegrationTest, SimpleFacetsTest, PrimitiveFieldTypeTest, AutoCommitTest, TestValueSourceCache, AnalyticsMergeStrategyTest, TestBinaryField, TestCloudInspectUtil, TestLRUCache, FullSolrCloudDistribCmdsTest, CSVRequestHandlerTest, TestDocBasedVersionConstraints, AddSchemaFieldsUpdateProcessorFactoryTest, DirectUpdateHandlerOptimizeTest, MigrateRouteKeyTest, MultiTermTest, SolrRequestParserTest, SampleTest, CustomCollectionTest, TestTolerantSearch, TestSolrXMLSerializer, OverseerRolesTest, TestSolr4Spatial, TestManagedSynonymFilterFactory, TestCopyFieldCollectionResource, TestUtils, TestGroupingSearch, HighlighterConfigTest, AbstractAnalyticsStatsTest, ResponseHeaderTest, MinimalSchemaTest, DistributedDebugComponentTest, TestManagedSchemaFieldResource, TestUniqueKeyFieldResource, DistributedExpandComponentTest, DateRangeFieldTest, TestSimpleQParserPlugin, AnalyticsQueryTest, RankQueryTest, ZkCLITest, TestSolrQueryParser, TestLuceneMatchVersion, TestStressUserVersions, TestSolrDeletionPolicy1, TermsComponentTest, SpellCheckComponentTest, RegexBoostProcessorTest, TestFieldSortValues, ResourceLoaderTest, TestCloudManagedSchema, AlternateDirectoryTest, DistributedTermsComponentTest, URLClassifyProcessorTest, ShardRoutingCustomTest, PolyFieldTest, DistributedSpellCheckComponentTest, EnumFieldTest, DeleteReplicaTest, HighlighterMaxOffsetTest, SpellingQueryConverterTest, DateMathParserTest, BadCopyFieldTest, TestComplexPhraseQParserPlugin, TestSweetSpotSimilarityFactory, TestExpandComponent, AssignTest, MultiThreadedOCPTest, TestCodecSupport, TestCursorMarkWithoutUniqueKey, DeleteShardTest, XmlUpdateRequestHandlerTest, InfoHandlerTest, SolrIndexSplitterTest, MergeStrategyTest, RangeFacetTest, FileBasedSpellCheckerTest, HighlighterTest, FieldFacetExtrasTest, RemoteQueryErrorTest, ExternalFileFieldSortTest, TestPostingsSolrHighlighter, TestManagedSchema, TestHighlightDedupGrouping, TestSolrXml, SolrPluginUtilsTest, UnloadDistributedZkTest, TestFastWriter, TestBM25SimilarityFactory, DocumentAnalysisRequestHandlerTest, SSLMigrationTest, OutputWriterTest, TestRTGBase, TestSolrJ, LukeRequestHandlerTest, SolrInfoMBeanTest, TestRequestStatusCollectionAPI, QueryParsingTest, TestSuggestSpellingConverter, TestNRTOpen, TestStressVersions, TimeZoneUtilsTest, ClusterStateUpdateTest, DistanceFunctionTest, CollectionsAPIAsyncDistributedZkTest, TestInitQParser, OverseerCollectionProcessorTest, ClusterStateTest, TestFuzzyAnalyzedSuggestions, NotRequiredUniqueKeyTest, TestNoOpRegenerator, CurrencyFieldXmlFileTest, TestCollationFieldDocValues, DeleteInactiveReplicaTest, TestFastLRUCache, TestStressRecovery, TestSolr4Spatial2, TestRangeQuery, DistribCursorPagingTest, TestSearcherReuse, TestHashPartitioner, TestDistributedGrouping, TestCoreDiscovery, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, ShardRoutingTest, BasicZkTest, TestReplicationHandler, TestRandomFaceting, ZkSolrClientTest, TestZkChroot, TestRandomDVFaceting, TestFaceting, TermVectorComponentDistributedTest, ZkControllerTest, TestRealTimeGet, TestReload, HardAutoCommitTest, TestCoreContainer, StatsComponentTest, QueryElevationComponentTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestSort, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, DirectUpdateHandlerTest, TestBadConfig, TestIndexSearcher, SoftAutoCommitTest, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, CoreAdminHandlerTest, DocValuesTest, SuggesterTSTTest, SuggesterTest, TestStressLucene, TestTrie, SpatialFilterTest, SuggesterWFSTTest, TestCSVLoader, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, DefaultValueUpdateProcessorTest, SortByFunctionTest, DocValuesMultiTest, XsltUpdateRequestHandlerTest, DebugComponentTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, StandardRequestHandlerTest, TestWriterPerf, TestReversedWildcardFilterFactory, TestOmitPositions, RequestHandlersTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, IndexSchemaTest, TestJmxIntegration, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, TestPHPSerializedResponseWriter, TestComponentsName, TestBinaryResponseWriter, TestQuerySenderListener, UpdateParamsTest, TestSolrIndexConfig, TestQuerySenderNoQuery, ResponseLogComponentTest, BadComponentTest, TestMergePolicyConfig, TestSolrDeletionPolicy2, TestDocSet, TestSearchPerf, NumericFieldsTest, TestConfig, TestSolrCoreProperties, TestPhraseSuggestions, SpellPossibilityIteratorTest, SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, DateFieldTest, DOMUtilTest, RAMDirectoryFactoryTest, TestDocumentBuilder, SystemInfoHandlerTest, UUIDFieldTest, FileUtilsTest, CircularListTest, DistributedIntervalFacetingTest, SolrTestCaseJ4Test, TestDistributedMissingSort, NoFacetTest, ExpressionTest, AbstractAnalyticsFacetTest, FieldFacetTest, QueryFacetTest, FunctionTest, AliasIntegrationTest, AsyncMigrateRouteKeyTest, ConnectionManagerTest, DistribDocExpirationUpdateProcessorTest, OverseerStatusTest, SolrXmlInZkTest, TestDistribDocBasedVersion, TestLeaderElectionZkExpiry, TestModifyConfFiles, TestShortCircuitedRequests, TriLevelCompositeIdRoutingTest, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, HdfsCollectionsAPIDistributedZkTest, HdfsRecoveryZkTest, HdfsSyncSliceTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest, QueryResultKeyTest, TestImplicitCoreProperties, TestInfoStreamLogging, TestShardHandlerFactory, TestSolrXmlPersistor, TestReplicationHandlerBackup, CoreAdminCreateDiscoverTest, CoreAdminRequestStatusTest, DistributedQueryComponentCustomSortTest, DistributedQueryComponentOptimizationTest, DistributedSuggestComponentTest, TestIntervalFaceting, TestChildDocTransformer, TestManagedResource, TestManagedResourceStorage, TestClassNameShortening, TestDynamicFieldResource, TestFieldCollectionResource, TestFieldResource, TestFieldTypeCollectionResource, TestSchemaNameResource, TestSchemaResource, TestSchemaVersionResource, TestSolrQueryParserResource, TestManagedStopFilterFactory, ChangedSchemaMergeTest, DocValuesMissingTest, ModifyConfFileTest, TestCloudManagedSchemaConcurrent, TestCloudSchemaless]
   [junit4] Completed in 283.67s, 1 test, 1 error <<< FAILURES!

[...truncated 69 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1298: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:922: There were test failures: 422 suites, 1761 tests, 1 error, 44 ignored (17 assumptions)

Total time: 162 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 1754 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1754/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.schema.TestCloudSchemaless.testDistribSearch

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:50726/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:50726/collection1
	at __randomizedtesting.SeedInfo.seed([ACB6DAD6A7E73D20:2D5054CED0B85D1C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:561)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
	at org.apache.solr.schema.TestCloudSchemaless.doTest(TestCloudSchemaless.java:140)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
	at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:911)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:449)
	... 46 more




Build Log:
[...truncated 11176 lines...]
   [junit4] Suite: org.apache.solr.schema.TestCloudSchemaless
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/init-core-data-001
   [junit4]   2> 2196466 T5463 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 2196467 T5463 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2196475 T5463 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 2196476 T5463 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2196478 T5464 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2196579 T5463 oasc.ZkTestServer.run start zk server on port:50715
   [junit4]   2> 2196582 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2196588 T5470 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69348b84 name:ZooKeeperConnection Watcher:127.0.0.1:50715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2196590 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2196590 T5463 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2196602 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2196620 T5472 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20103ea7 name:ZooKeeperConnection Watcher:127.0.0.1:50715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2196621 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2196621 T5463 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2196627 T5463 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2196634 T5463 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2196641 T5463 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2196650 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-schemaless.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2196650 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2196662 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema-add-schema-fields-update-processor.xml to /configs/conf1/schema.xml
   [junit4]   2> 2196663 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2196673 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2196673 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2196682 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2196683 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2196691 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2196692 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2196699 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2196701 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2196710 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2196710 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2196719 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2196720 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2196728 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2196729 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2196737 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2196738 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2196748 T5463 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2196749 T5463 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2196761 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2196766 T5474 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@332219b2 name:ZooKeeperConnection Watcher:127.0.0.1:50715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2196767 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2198700 T5463 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2198709 T5463 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2198728 T5463 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50719
   [junit4]   2> 2198733 T5463 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2198733 T5463 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2198734 T5463 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002
   [junit4]   2> 2198735 T5463 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/'
   [junit4]   2> 2198803 T5463 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/solr.xml
   [junit4]   2> 2198835 T5463 oasc.CoreContainer.<init> New CoreContainer 896101336
   [junit4]   2> 2198836 T5463 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/]
   [junit4]   2> 2198838 T5463 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2198838 T5463 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2198838 T5463 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2198838 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2198839 T5463 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2198839 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2198839 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2198840 T5463 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2198840 T5463 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2198844 T5463 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2198846 T5463 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2198846 T5463 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2198847 T5463 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2198847 T5463 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50715/solr
   [junit4]   2> 2198848 T5463 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2198849 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2198855 T5485 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@239af761 name:ZooKeeperConnection Watcher:127.0.0.1:50715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2198858 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2198864 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2198871 T5487 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ecac93c name:ZooKeeperConnection Watcher:127.0.0.1:50715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2198872 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2198879 T5463 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2198888 T5463 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2198899 T5463 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 2198911 T5463 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 2198921 T5463 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 2198935 T5463 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2198940 T5463 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50719_
   [junit4]   2> 2198944 T5463 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50719_
   [junit4]   2> 2198956 T5463 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2198961 T5463 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2198971 T5463 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50719_
   [junit4]   2> 2198972 T5463 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2198977 T5463 oasc.Overseer.start Overseer (id=92231335358824452-127.0.0.1:50719_-n_0000000000) starting
   [junit4]   2> 2198985 T5463 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2199013 T5489 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2199014 T5463 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2199020 T5463 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2199024 T5463 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2199033 T5488 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2199040 T5490 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 2199040 T5490 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2199042 T5490 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2199043 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2199047 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2199049 T5488 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50719",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50719_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2199049 T5488 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2199049 T5488 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2199055 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2199057 T5491 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2200045 T5490 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2200046 T5490 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2200047 T5490 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2200048 T5490 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2200051 T5490 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2200051 T5490 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/'
   [junit4]   2> 2200053 T5490 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 2200053 T5490 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 2200054 T5490 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 2200099 T5490 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2200106 T5490 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2200109 T5490 oass.ManagedIndexSchemaFactory.create The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
   [junit4]   2> 2200112 T5490 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2200115 T5490 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 2200125 T5490 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2200128 T5490 oass.ManagedIndexSchema.persistManagedSchemaToZooKeeper Created and persisted managed schema znode at /configs/conf1/managed-schema
   [junit4]   2> 2200133 T5490 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml.bak
   [junit4]   2> 2200144 T5490 oass.ManagedIndexSchemaFactory.zkUgradeToManagedSchema After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/conf1/schema.xml to /configs/conf1/schema.xml.bak
   [junit4]   2> 2200144 T5490 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2200145 T5490 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 2200145 T5490 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/data/
   [junit4]   2> 2200145 T5490 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 2200146 T5490 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/data
   [junit4]   2> 2200147 T5490 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/data/index/
   [junit4]   2> 2200148 T5490 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2200148 T5490 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/data/index
   [junit4]   2> 2200148 T5490 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=34.16796875, floorSegmentMB=1.3173828125, forceMergeDeletesPctAllowed=24.861193504378168, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6822459123050819
   [junit4]   2> 2200156 T5490 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@226f398e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2200156 T5490 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2200158 T5490 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 2200161 T5490 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2200162 T5490 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2200163 T5490 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2200163 T5490 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2200163 T5490 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2200163 T5490 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2200164 T5490 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2200164 T5490 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2200164 T5490 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2200164 T5490 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2200165 T5490 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2200165 T5490 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2200173 T5490 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2200174 T5490 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2200176 T5490 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2200177 T5490 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2200179 T5490 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2200181 T5490 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2200181 T5490 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2200182 T5490 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=93.9990234375, floorSegmentMB=1.8115234375, forceMergeDeletesPctAllowed=14.888475819124274, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6231248174670746
   [junit4]   2> 2200183 T5490 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-002/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@226f398e; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2200184 T5490 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2200184 T5490 oass.SolrIndexSearcher.<init> Opening Searcher@50266db5[collection1] main
   [junit4]   2> 2200184 T5490 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2200187 T5490 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2200187 T5490 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2200189 T5490 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2200189 T5490 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2200189 T5490 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2200191 T5490 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2200191 T5490 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2200193 T5490 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
   [junit4]   2> 2200200 T5490 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2200200 T5490 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2200200 T5490 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2200200 T5490 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 2200203 T5490 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2200204 T5492 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@50266db5[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2200206 T5495 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50719 collection:control_collection shard:shard1
   [junit4]   2> 2200207 T5463 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2200207 T5463 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2200208 T5495 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2200214 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2200218 T5497 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38222a1a name:ZooKeeperConnection Watcher:127.0.0.1:50715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2200219 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2200223 T5463 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2200223 T5495 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2200228 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2200229 T5463 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2200229 T5495 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2200230 T5495 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2251 name=collection1 org.apache.solr.core.SolrCore@7093603b url=https://127.0.0.1:50719/collection1 node=127.0.0.1:50719_ C2251_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:50719, core=collection1, node_name=127.0.0.1:50719_}
   [junit4]   2> 2200230 T5495 C2251 P50719 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50719/collection1/
   [junit4]   2> 2200230 T5495 C2251 P50719 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2200231 T5495 C2251 P50719 oasc.SyncStrategy.syncToMe https://127.0.0.1:50719/collection1/ has no replicas
   [junit4]   2> 2200231 T5495 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50719/collection1/ shard1
   [junit4]   2> 2200231 T5495 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2200234 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2200243 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2200249 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2200257 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2200364 T5491 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2200366 T5498 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2200405 T5495 oasc.ZkController.register We are https://127.0.0.1:50719/collection1/ and leader is https://127.0.0.1:50719/collection1/
   [junit4]   2> 2200406 T5495 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50719
   [junit4]   2> 2200406 T5495 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2200406 T5495 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 2200406 T5495 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2200410 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2200415 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2200415 T5499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2200419 T5495 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2200424 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2200425 T5488 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50719",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50719_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2200432 T5499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2200540 T5498 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2200541 T5499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2202129 T5463 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2202137 T5463 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2202153 T5463 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2202157 T5463 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50723
   [junit4]   2> 2202160 T5463 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2202160 T5463 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2202161 T5463 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003
   [junit4]   2> 2202161 T5463 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/'
   [junit4]   2> 2202206 T5463 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/solr.xml
   [junit4]   2> 2202223 T5463 oasc.CoreContainer.<init> New CoreContainer 2096131068
   [junit4]   2> 2202223 T5463 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/]
   [junit4]   2> 2202224 T5463 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2202224 T5463 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2202224 T5463 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2202225 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2202225 T5463 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2202225 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2202226 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2202226 T5463 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2202227 T5463 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2202230 T5463 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2202231 T5463 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2202232 T5463 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2202232 T5463 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2202232 T5463 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50715/solr
   [junit4]   2> 2202233 T5463 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2202234 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2202238 T5510 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e67bddd name:ZooKeeperConnection Watcher:127.0.0.1:50715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2202239 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2202244 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2202247 T5512 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5926e72c name:ZooKeeperConnection Watcher:127.0.0.1:50715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2202248 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2202267 T5463 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2203277 T5463 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50723_
   [junit4]   2> 2203279 T5463 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50723_
   [junit4]   2> 2203302 T5514 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2203302 T5514 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2203304 T5514 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2203304 T5499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2203305 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2203306 T5515 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2203309 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2203310 T5488 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50723",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50723_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2203310 T5488 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 2203311 T5488 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2203315 T5515 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2203318 T5498 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2203320 T5515 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2203324 T5513 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2204306 T5514 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2204307 T5514 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2204309 T5514 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2204309 T5514 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2204311 T5514 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2204311 T5514 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/'
   [junit4]   2> 2204313 T5514 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 2204314 T5514 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 2204315 T5514 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 2204371 T5514 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2204383 T5514 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2204388 T5514 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 2204392 T5514 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 2204403 T5514 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2204405 T5514 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2204406 T5514 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 2204407 T5514 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/data/
   [junit4]   2> 2204407 T5514 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 2204408 T5514 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/data
   [junit4]   2> 2204409 T5514 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/data/index/
   [junit4]   2> 2204411 T5514 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2204412 T5514 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/data/index
   [junit4]   2> 2204412 T5514 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=34.16796875, floorSegmentMB=1.3173828125, forceMergeDeletesPctAllowed=24.861193504378168, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6822459123050819
   [junit4]   2> 2204418 T5514 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@bcaf984; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2204418 T5514 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2204420 T5514 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 2204421 T5514 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2204421 T5514 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2204421 T5514 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2204422 T5514 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2204422 T5514 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2204423 T5514 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2204423 T5514 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2204423 T5514 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2204424 T5514 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2204436 T5514 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2204436 T5514 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2204436 T5514 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2204445 T5514 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2204447 T5514 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2204449 T5514 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2204451 T5514 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2204453 T5514 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2204454 T5514 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2204455 T5514 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2204457 T5514 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=93.9990234375, floorSegmentMB=1.8115234375, forceMergeDeletesPctAllowed=14.888475819124274, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6231248174670746
   [junit4]   2> 2204460 T5514 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-003/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@bcaf984; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2204461 T5514 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2204461 T5514 oass.SolrIndexSearcher.<init> Opening Searcher@5e37642a[collection1] main
   [junit4]   2> 2204461 T5514 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2204466 T5514 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2204466 T5514 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2204468 T5514 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2204469 T5514 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2204469 T5514 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2204473 T5514 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2204473 T5514 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2204473 T5514 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2204474 T5514 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2204474 T5514 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 2204481 T5514 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2204481 T5516 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5e37642a[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2204483 T5519 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50723 collection:collection1 shard:shard2
   [junit4]   2> 2204484 T5463 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2204485 T5463 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2204486 T5519 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2204507 T5519 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2204514 T5519 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2204515 T5519 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2252 name=collection1 org.apache.solr.core.SolrCore@1a31ddcb url=https://127.0.0.1:50723/collection1 node=127.0.0.1:50723_ C2252_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50723, core=collection1, node_name=127.0.0.1:50723_}
   [junit4]   2> 2204515 T5519 C2252 P50723 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50723/collection1/
   [junit4]   2> 2204516 T5519 C2252 P50723 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2204517 T5519 C2252 P50723 oasc.SyncStrategy.syncToMe https://127.0.0.1:50723/collection1/ has no replicas
   [junit4]   2> 2204517 T5519 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50723/collection1/ shard2
   [junit4]   2> 2204519 T5519 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2204517 T5515 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2204534 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2204546 T5515 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2204567 T5515 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2204673 T5498 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2204674 T5515 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2204675 T5513 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2204709 T5519 oasc.ZkController.register We are https://127.0.0.1:50723/collection1/ and leader is https://127.0.0.1:50723/collection1/
   [junit4]   2> 2204709 T5519 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50723
   [junit4]   2> 2204710 T5519 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2204710 T5519 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2204710 T5519 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2204722 T5519 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2204723 T5515 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2204724 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2204727 T5499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2204732 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2204734 T5488 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50723",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50723_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2204743 T5499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2204850 T5499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2204852 T5513 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2204853 T5498 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2206610 T5463 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2206612 T5463 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2206621 T5463 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2206623 T5463 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50726
   [junit4]   2> 2206627 T5463 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2206628 T5463 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2206628 T5463 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004
   [junit4]   2> 2206629 T5463 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/'
   [junit4]   2> 2206671 T5463 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/solr.xml
   [junit4]   2> 2206687 T5463 oasc.CoreContainer.<init> New CoreContainer 583264522
   [junit4]   2> 2206688 T5463 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/]
   [junit4]   2> 2206688 T5463 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2206689 T5463 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2206689 T5463 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2206690 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2206690 T5463 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2206690 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2206691 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2206691 T5463 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2206692 T5463 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2206695 T5463 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=false
   [junit4]   2> 2206697 T5463 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2206697 T5463 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2206698 T5463 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2206698 T5463 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50715/solr
   [junit4]   2> 2206699 T5463 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2206702 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2206706 T5530 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f524789 name:ZooKeeperConnection Watcher:127.0.0.1:50715 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2206707 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2206714 T5463 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2206717 T5532 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8d56f35 name:ZooKeeperConnection Watcher:127.0.0.1:50715/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2206718 T5463 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2206738 T5463 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2207748 T5463 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50726_
   [junit4]   2> 2207752 T5463 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50726_
   [junit4]   2> 2207778 T5534 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 2207779 T5534 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2207781 T5534 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2207781 T5499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2207781 T5515 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2207781 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2207787 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2207789 T5488 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50726",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50726_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2207789 T5488 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=4
   [junit4]   2> 2207790 T5488 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 2207796 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2207798 T5491 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2207799 T5513 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2207798 T5498 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2207798 T5533 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2208783 T5534 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2208784 T5534 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2208785 T5534 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2208786 T5534 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2208789 T5534 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2208790 T5534 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/'
   [junit4]   2> 2208791 T5534 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 2208793 T5534 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 2208793 T5534 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 2208868 T5534 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
   [junit4]   2> 2208875 T5534 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2208879 T5534 oass.IndexSchema.readSchema Reading Solr Schema from managed-schema
   [junit4]   2> 2208883 T5534 oass.IndexSchema.readSchema [collection1] Schema name=add-schema-fields-update-processor
   [junit4]   2> 2208917 T5534 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2208917 T5534 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 2208917 T5534 oasc.SolrCore.initDirectoryFactory solr.NRTCachingDirectoryFactory
   [junit4]   2> 2208918 T5534 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/, dataDir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/data/
   [junit4]   2> 2208918 T5534 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 2208919 T5534 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/data
   [junit4]   2> 2208920 T5534 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/data/index/
   [junit4]   2> 2208922 T5534 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2208923 T5534 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/data/index
   [junit4]   2> 2208924 T5534 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=27, maxMergeAtOnceExplicit=20, maxMergedSegmentMB=34.16796875, floorSegmentMB=1.3173828125, forceMergeDeletesPctAllowed=24.861193504378168, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6822459123050819
   [junit4]   2> 2208930 T5534 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7ea2b3d4; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2208931 T5534 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2208934 T5534 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "add-unknown-fields-to-the-schema"
   [junit4]   2> 2208935 T5534 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2208935 T5534 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2208936 T5534 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2208936 T5534 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2208936 T5534 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2208937 T5534 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2208938 T5534 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2208938 T5534 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2208938 T5534 oasc.RequestHandlers.initHandlersFromConfig WARN Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 2208938 T5534 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2208938 T5534 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2208939 T5534 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2208945 T5534 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2208948 T5534 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2208949 T5534 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2208950 T5534 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2208951 T5534 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2208955 T5534 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2208955 T5534 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2208956 T5534 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=40, maxMergedSegmentMB=93.9990234375, floorSegmentMB=1.8115234375, forceMergeDeletesPctAllowed=14.888475819124274, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6231248174670746
   [junit4]   2> 2208959 T5534 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=NRTCachingDirectory(MMapDirectory@/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-004/collection1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7ea2b3d4; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
   [junit4]   2> 2208959 T5534 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2208959 T5534 oass.SolrIndexSearcher.<init> Opening Searcher@7cee7431[collection1] main
   [junit4]   2> 2208960 T5534 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2208963 T5534 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 2208963 T5534 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2208964 T5534 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2208965 T5534 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 2208965 T5534 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2208968 T5534 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2208968 T5534 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2208968 T5534 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
   [junit4]   2> 2208968 T5534 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 2208968 T5534 oass.ZkIndexSchemaReader.createSchemaWatcher Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema ...
   [junit4]   2> 2208972 T5534 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2208972 T5535 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7cee7431[collection1] main{UninvertingDirectoryReader()}
   [junit4]   2> 2208975 T5538 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:50726 collection:collection1 shard:shard3
   [junit4]   2> 2208977 T5463 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2208977 T5463 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2208978 T5538 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 2208989 T5538 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 2208993 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2208998 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2208999 T5538 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2208999 T5538 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2253 name=collection1 org.apache.solr.core.SolrCore@6e5c6456 url=https://127.0.0.1:50726/collection1 node=127.0.0.1:50726_ C2253_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:50726, core=collection1, node_name=127.0.0.1:50726_}
   [junit4]   2> 2208999 T5538 C2253 P50726 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:50726/collection1/
   [junit4]   2> 2208999 T5538 C2253 P50726 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2209000 T5538 C2253 P50726 oasc.SyncStrategy.syncToMe https://127.0.0.1:50726/collection1/ has no replicas
   [junit4]   2> 2209000 T5538 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:50726/collection1/ shard3
   [junit4]   2> 2209000 T5538 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 2209019 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2209027 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2209037 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2209142 T5498 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2209145 T5533 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2209146 T5491 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2209148 T5513 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2209180 T5538 oasc.ZkController.register We are https://127.0.0.1:50726/collection1/ and leader is https://127.0.0.1:50726/collection1/
   [junit4]   2> 2209181 T5538 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:50726
   [junit4]   2> 2209181 T5538 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2209181 T5538 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 2209181 T5538 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2209185 T5491 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2209185 T5515 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2209186 T5499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2209187 T5538 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2209193 T5488 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2209194 T5488 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"https://127.0.0.1:50726",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:50726_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2209198 T5499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2209305 T5533 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2209305 T5513 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2209307 T5499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2209307 T5498 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2210850 T5463 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2210851 T5463 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2210863 T5463 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
   [junit4]   2> 2210863 T5463 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:50729
   [junit4]   2> 2210864 T5463 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2210865 T5463 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2210868 T5463 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-005
   [junit4]   2> 2210869 T5463 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-005/'
   [junit4]   2> 2210922 T5463 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-005/solr.xml
   [junit4]   2> 2210940 T5463 oasc.CoreContainer.<init> New CoreContainer 137718792
   [junit4]   2> 2210940 T5463 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001/tempDir-005/]
   [junit4]   2> 2210941 T5463 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2210941 T5463 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 2210941 T5463 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2210942 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2210942 T5463 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2210942 T5463 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2210943 T5463 oashc.HttpSha

[...truncated too long message...]

 [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCloudSchemaless -Dtests.method=testDistribSearch -Dtests.seed=ACB6DAD6A7E73D20 -Dtests.slow=true -Dtests.locale=ar_TN -Dtests.timezone=America/Marigot -Dtests.file.encoding=UTF-8
   [junit4] ERROR    162s | TestCloudSchemaless.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:50726/collection1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ACB6DAD6A7E73D20:2D5054CED0B85D1C]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:561)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206)
   [junit4]    > 	at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
   [junit4]    > 	at org.apache.solr.schema.TestCloudSchemaless.doTest(TestCloudSchemaless.java:140)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:865)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.net.SocketTimeoutException: Read timed out
   [junit4]    > 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:150)
   [junit4]    > 	at java.net.SocketInputStream.read(SocketInputStream.java:121)
   [junit4]    > 	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
   [junit4]    > 	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954)
   [junit4]    > 	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:911)
   [junit4]    > 	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
   [junit4]    > 	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:682)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:449)
   [junit4]    > 	... 46 more
   [junit4]   2> 2358899 T5463 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./temp/solr.schema.TestCloudSchemaless-ACB6DAD6A7E73D20-001
   [junit4]   2> 162440 T5462 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=ar_TN, timezone=America/Marigot
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.8.0_11 (64-bit)/cpus=2,threads=1,free=148631104,total=262209536
   [junit4]   2> NOTE: All tests run in this JVM: [TestRemoteStreaming, CSVRequestHandlerTest, DisMaxRequestHandlerTest, BlockDirectoryTest, TestSearcherReuse, TestBinaryField, ClusterStateTest, DistributedQueryComponentOptimizationTest, ExpressionTest, AliasIntegrationTest, DirectSolrSpellCheckerTest, BadComponentTest, DirectUpdateHandlerTest, URLClassifyProcessorTest, QueryResultKeyTest, StressHdfsTest, SuggestComponentTest, TriLevelCompositeIdRoutingTest, TestCharFilters, FieldMutatingUpdateProcessorTest, TestDynamicFieldResource, SolrIndexConfigTest, CoreAdminCreateDiscoverTest, LoggingHandlerTest, PolyFieldTest, LukeRequestHandlerTest, TestTolerantSearch, TestStressRecovery, TestPerFieldSimilarity, StatsComponentTest, TestSolrDeletionPolicy1, ZkSolrClientTest, ReplicationFactorTest, TestCollationFieldDocValues, ShardRoutingCustomTest, TestCollationField, TestSolrCoreProperties, MigrateRouteKeyTest, TestConfigSets, TestGroupingSearch, ParsingFieldUpdateProcessorsTest, MinimalSchemaTest, SuggesterTSTTest, TestSimpleQParserPlugin, TestDistributedGrouping, DistribDocExpirationUpdateProcessorTest, SampleTest, ScriptEngineTest, TestStandardQParsers, TestFieldTypeResource, HdfsUnloadDistributedZkTest, BadIndexSchemaTest, TestOmitPositions, TestReversedWildcardFilterFactory, NumericFieldsTest, TestAddFieldRealTimeGet, SOLR749Test, FullSolrCloudDistribCmdsTest, ResponseLogComponentTest, CoreAdminHandlerTest, TestBM25SimilarityFactory, TestFileDictionaryLookup, TestFieldSortValues, SolrIndexSplitterTest, CurrencyFieldXmlFileTest, TestSolrDeletionPolicy2, AnalyticsMergeStrategyTest, CustomCollectionTest, TestSchemaResource, TestMissingGroups, TestIndexSearcher, TestModifyConfFiles, UniqFieldsUpdateProcessorFactoryTest, TestBinaryResponseWriter, UpdateRequestProcessorFactoryTest, TestCursorMarkWithoutUniqueKey, TestMaxScoreQueryParser, TestElisionMultitermQuery, MultiTermTest, SolrInfoMBeanTest, TestFreeTextSuggestions, TestExtendedDismaxParser, SimplePostToolTest, TestSolrXMLSerializer, AlternateDirectoryTest, SuggesterTest, HardAutoCommitTest, StandardRequestHandlerTest, TestIBSimilarityFactory, TestRandomFaceting, TestCollectionAPI, UUIDFieldTest, PrimitiveFieldTypeTest, UnloadDistributedZkTest, CursorMarkTest, EnumFieldTest, InfoHandlerTest, TestPseudoReturnFields, TestQueryUtils, TestSolrXml, ConnectionManagerTest, StatelessScriptUpdateProcessorFactoryTest, IndexBasedSpellCheckerTest, TestCSVResponseWriter, TestReload, TestInfoStreamLogging, IndexSchemaTest, ReturnFieldsTest, TestPartialUpdateDeduplication, BasicDistributedZkTest, TestXIncludeConfig, DistributedIntervalFacetingTest, TestDefaultSearchFieldResource, TestBlendedInfixSuggestions, TestCoreContainer, IndexSchemaRuntimeFieldTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSort, SimpleFacetsTest, TestFunctionQuery, TestDocBasedVersionConstraints, TestAtomicUpdateErrorCases, SoftAutoCommitTest, DocumentAnalysisRequestHandlerTest, HdfsWriteToMultipleCollectionsTest, TestPHPSerializedResponseWriter, DOMUtilTest, ChaosMonkeyNothingIsSafeTest, TestStressVersions, DirectSolrConnectionTest, TestFastLRUCache, PreAnalyzedUpdateProcessorTest, JSONWriterTest, CollectionsAPIAsyncDistributedZkTest, AnalysisErrorHandlingTest, OutputWriterTest, BasicFunctionalityTest, TestCoreDiscovery, DeleteReplicaTest, TestIndexingPerformance, TestCloudManagedSchemaConcurrent, TestMergePolicyConfig, TestUniqueKeyFieldResource, RequiredFieldsTest, TestSolrIndexConfig, BlockCacheTest, TestReplicationHandlerBackup, OpenExchangeRatesOrgProviderTest, SpellPossibilityIteratorTest, TestCloudSchemaless]
   [junit4] Completed in 162.98s, 1 test, 1 error <<< FAILURES!

[...truncated 835 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1298: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:922: There were test failures: 422 suites, 1757 tests, 1 error, 44 ignored (17 assumptions)

Total time: 151 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any