You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/06/06 03:51:45 UTC

[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 1589 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/1589/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC

3 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([D0D8F82F992F0231]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([D0D8F82F992F0231]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Captured an uncaught exception in thread: Thread[id=1876, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1876, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
Caused by: java.lang.AssertionError: liveDocs.count()=6 info.docCount=10 info.getDelCount()=5
	at __randomizedtesting.SeedInfo.seed([D0D8F82F992F0231]:0)
	at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:92)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:110)
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:237)
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:98)
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:393)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:273)
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1471)
	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1672)
	at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:673)
	at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:493)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)




Build Log:
[...truncated 11201 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/init-core-data-001
   [junit4]   2> 474956 T1504 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 474957 T1504 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /oyxj/
   [junit4]   2> 474964 T1504 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 474967 T1504 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 474968 T1505 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 475068 T1504 oasc.ZkTestServer.run start zk server on port:54351
   [junit4]   2> 475070 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 475078 T1511 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@21505582 name:ZooKeeperConnection Watcher:127.0.0.1:54351 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 475079 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 475079 T1504 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 475099 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 475104 T1513 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e108a72 name:ZooKeeperConnection Watcher:127.0.0.1:54351/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 475104 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 475105 T1504 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 475116 T1504 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 475124 T1504 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 475133 T1504 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 475141 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 475141 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 475154 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 475155 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 475164 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 475164 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 475177 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 475177 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 475186 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 475187 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 475196 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 475197 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 475206 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 475206 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 475215 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 475216 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 475225 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 475225 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 475235 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 475236 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 475244 T1504 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 475245 T1504 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 476984 T1504 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 476994 T1504 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54354
   [junit4]   2> 476996 T1504 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 476996 T1504 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 476997 T1504 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002
   [junit4]   2> 476997 T1504 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002/'
   [junit4]   2> 477047 T1504 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002/solr.xml
   [junit4]   2> 477138 T1504 oasc.CoreContainer.<init> New CoreContainer 599677777
   [junit4]   2> 477138 T1504 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002/]
   [junit4]   2> 477141 T1504 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 477141 T1504 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 477142 T1504 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 477142 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 477142 T1504 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 477142 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 477143 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 477143 T1504 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 477143 T1504 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 477144 T1504 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 477145 T1504 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 477145 T1504 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 477145 T1504 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54351/solr
   [junit4]   2> 477146 T1504 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 477147 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 477152 T1524 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@265d352c name:ZooKeeperConnection Watcher:127.0.0.1:54351 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 477154 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 477160 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 477164 T1526 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fb815b name:ZooKeeperConnection Watcher:127.0.0.1:54351/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 477165 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 477169 T1504 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 477181 T1504 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 477193 T1504 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 477207 T1504 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 477217 T1504 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 477229 T1504 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 477234 T1504 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54354_oyxj
   [junit4]   2> 477236 T1504 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54354_oyxj
   [junit4]   2> 477245 T1504 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 477252 T1504 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 477265 T1504 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:54354_oyxj
   [junit4]   2> 477265 T1504 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 477272 T1504 oasc.Overseer.start Overseer (id=91882267053326339-127.0.0.1:54354_oyxj-n_0000000000) starting
   [junit4]   2> 477280 T1504 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 477313 T1528 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 477315 T1504 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 477321 T1504 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 477327 T1504 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 477336 T1527 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 477345 T1529 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 477345 T1529 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 477348 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477348 T1529 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 477353 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 477355 T1527 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54354/oyxj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54354_oyxj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 477355 T1527 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 477356 T1527 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 477364 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 477367 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 478351 T1529 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 478351 T1529 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 478353 T1529 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 478353 T1529 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 478357 T1529 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 478357 T1529 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002/collection1/'
   [junit4]   2> 478359 T1529 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 478360 T1529 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 478362 T1529 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 478447 T1529 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 478505 T1529 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 478510 T1529 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 478525 T1529 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 479011 T1529 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 479014 T1529 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 479014 T1529 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 479024 T1529 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 479104 T1529 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 479105 T1529 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 479105 T1529 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-002/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/control/data/
   [junit4]   2> 479105 T1529 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2539efe5
   [junit4]   2> 479107 T1529 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/control/data
   [junit4]   2> 479108 T1529 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/control/data/index/
   [junit4]   2> 479109 T1529 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 479110 T1529 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/control/data/index
   [junit4]   2> 479111 T1529 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=79.1689453125, floorSegmentMB=0.3916015625, forceMergeDeletesPctAllowed=14.097871798244634, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 479116 T1529 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 479117 T1529 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 479123 T1529 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 479124 T1529 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 479125 T1529 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 479125 T1529 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 479125 T1529 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 479126 T1529 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 479126 T1529 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 479127 T1529 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 479127 T1529 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 479128 T1529 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 479129 T1529 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 479130 T1529 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 479131 T1529 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 479131 T1529 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 479132 T1529 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 479132 T1529 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 479148 T1529 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 479153 T1529 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 479153 T1529 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 479154 T1529 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=24.1015625, floorSegmentMB=0.21875, forceMergeDeletesPctAllowed=24.631939252862697, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3203946024898354
   [junit4]   2> 479157 T1529 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 479157 T1529 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 479158 T1529 oass.SolrIndexSearcher.<init> Opening Searcher@5c031fed[collection1] main
   [junit4]   2> 479158 T1529 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 479161 T1529 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 479161 T1529 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 479163 T1529 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 479163 T1529 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 479163 T1529 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 479165 T1529 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 479165 T1529 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 479165 T1529 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 479170 T1529 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 479174 T1530 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5c031fed[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 479175 T1529 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 479179 T1533 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54354/oyxj collection:control_collection shard:shard1
   [junit4]   2> 479180 T1504 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 479181 T1504 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 479182 T1533 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 479185 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 479191 T1535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@91c108d name:ZooKeeperConnection Watcher:127.0.0.1:54351/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 479191 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 479194 T1504 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 479202 T1504 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 479205 T1533 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 479211 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479214 T1533 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 479214 T1533 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C789 name=collection1 org.apache.solr.core.SolrCore@44207aaf url=http://127.0.0.1:54354/oyxj/collection1 node=127.0.0.1:54354_oyxj C789_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:54354/oyxj, core=collection1, node_name=127.0.0.1:54354_oyxj}
   [junit4]   2> 479215 T1533 C789 P54354 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54354/oyxj/collection1/
   [junit4]   2> 479216 T1533 C789 P54354 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 479216 T1533 C789 P54354 oasc.SyncStrategy.syncToMe http://127.0.0.1:54354/oyxj/collection1/ has no replicas
   [junit4]   2> 479216 T1533 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54354/oyxj/collection1/ shard1
   [junit4]   2> 479217 T1533 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 479220 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479234 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479239 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 479240 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 479257 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479267 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479373 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 479373 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 479419 T1533 oasc.ZkController.register We are http://127.0.0.1:54354/oyxj/collection1/ and leader is http://127.0.0.1:54354/oyxj/collection1/
   [junit4]   2> 479419 T1533 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54354/oyxj
   [junit4]   2> 479419 T1533 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 479420 T1533 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 479420 T1533 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 479426 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479427 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479428 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479432 T1533 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479435 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 479437 T1527 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54354/oyxj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54354_oyxj",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 479447 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 479557 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 479558 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 481612 T1504 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 481614 T1504 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 481622 T1504 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54358
   [junit4]   2> 481623 T1504 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 481624 T1504 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 481631 T1504 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003
   [junit4]   2> 481631 T1504 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003/'
   [junit4]   2> 481718 T1504 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003/solr.xml
   [junit4]   2> 481860 T1504 oasc.CoreContainer.<init> New CoreContainer 1971847240
   [junit4]   2> 481864 T1504 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003/]
   [junit4]   2> 481870 T1504 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 481871 T1504 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 481872 T1504 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 481873 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 481873 T1504 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 481874 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 481874 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 481875 T1504 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 481876 T1504 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 481877 T1504 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 481879 T1504 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 481879 T1504 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 481883 T1504 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54351/solr
   [junit4]   2> 481884 T1504 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 481886 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 481896 T1546 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33cab340 name:ZooKeeperConnection Watcher:127.0.0.1:54351 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 481898 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 481908 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 481917 T1548 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42dba59 name:ZooKeeperConnection Watcher:127.0.0.1:54351/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 481921 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 481957 T1504 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 482981 T1504 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54358_oyxj
   [junit4]   2> 482984 T1504 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54358_oyxj
   [junit4]   2> 482995 T1526 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 482995 T1535 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 482995 T1548 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 483018 T1549 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 483018 T1549 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 483020 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 483021 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 483021 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 483020 T1549 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 483025 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 483027 T1527 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54358/oyxj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54358_oyxj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 483027 T1527 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
   [junit4]   2> 483027 T1527 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 483035 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 483038 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 483038 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 483038 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 484023 T1549 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 484024 T1549 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 484025 T1549 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 484026 T1549 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 484029 T1549 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 484030 T1549 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003/collection1/'
   [junit4]   2> 484034 T1549 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 484035 T1549 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 484036 T1549 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 484127 T1549 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 484178 T1549 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 484181 T1549 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 484190 T1549 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 484674 T1549 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 484677 T1549 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 484680 T1549 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 484687 T1549 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 484768 T1549 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 484769 T1549 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 484769 T1549 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-003/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty1/
   [junit4]   2> 484769 T1549 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2539efe5
   [junit4]   2> 484770 T1549 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty1
   [junit4]   2> 484771 T1549 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty1/index/
   [junit4]   2> 484771 T1549 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 484772 T1549 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty1/index
   [junit4]   2> 484773 T1549 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=79.1689453125, floorSegmentMB=0.3916015625, forceMergeDeletesPctAllowed=14.097871798244634, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 484776 T1549 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 484777 T1549 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 484784 T1549 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 484786 T1549 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 484786 T1549 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 484787 T1549 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 484787 T1549 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 484787 T1549 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 484788 T1549 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 484788 T1549 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 484789 T1549 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 484790 T1549 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 484791 T1549 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 484791 T1549 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 484791 T1549 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 484792 T1549 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 484793 T1549 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 484794 T1549 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 484806 T1549 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 484809 T1549 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 484809 T1549 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 484811 T1549 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=24.1015625, floorSegmentMB=0.21875, forceMergeDeletesPctAllowed=24.631939252862697, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3203946024898354
   [junit4]   2> 484813 T1549 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 484813 T1549 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 484813 T1549 oass.SolrIndexSearcher.<init> Opening Searcher@4543aeb[collection1] main
   [junit4]   2> 484814 T1549 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 484818 T1549 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 484818 T1549 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 484821 T1549 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 484821 T1549 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 484821 T1549 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 484823 T1549 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 484823 T1549 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 484824 T1549 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 484824 T1549 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 484829 T1550 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4543aeb[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 484832 T1549 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 484834 T1553 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54358/oyxj collection:collection1 shard:shard1
   [junit4]   2> 484835 T1504 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 484835 T1504 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 484836 T1553 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 484855 T1553 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 484868 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 484874 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 484877 T1553 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 484877 T1553 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C790 name=collection1 org.apache.solr.core.SolrCore@3a1cbc10 url=http://127.0.0.1:54358/oyxj/collection1 node=127.0.0.1:54358_oyxj C790_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:54358/oyxj, core=collection1, node_name=127.0.0.1:54358_oyxj}
   [junit4]   2> 484878 T1553 C790 P54358 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54358/oyxj/collection1/
   [junit4]   2> 484879 T1553 C790 P54358 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 484879 T1553 C790 P54358 oasc.SyncStrategy.syncToMe http://127.0.0.1:54358/oyxj/collection1/ has no replicas
   [junit4]   2> 484879 T1553 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54358/oyxj/collection1/ shard1
   [junit4]   2> 484879 T1553 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 484885 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 484890 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 484890 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 484892 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 484907 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 484914 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 484923 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 485031 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 485032 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 485032 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 485067 T1553 oasc.ZkController.register We are http://127.0.0.1:54358/oyxj/collection1/ and leader is http://127.0.0.1:54358/oyxj/collection1/
   [junit4]   2> 485067 T1553 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54358/oyxj
   [junit4]   2> 485068 T1553 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 485068 T1553 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 485068 T1553 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 485073 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 485073 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 485073 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 485075 T1553 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 485079 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 485082 T1527 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54358/oyxj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54358_oyxj",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 485090 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 485201 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 485204 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 485205 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 486569 T1504 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 486570 T1504 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 486578 T1504 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54361
   [junit4]   2> 486579 T1504 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 486580 T1504 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 486581 T1504 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004
   [junit4]   2> 486581 T1504 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004/'
   [junit4]   2> 486629 T1504 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004/solr.xml
   [junit4]   2> 486706 T1504 oasc.CoreContainer.<init> New CoreContainer 1698645276
   [junit4]   2> 486706 T1504 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004/]
   [junit4]   2> 486707 T1504 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 486708 T1504 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 486708 T1504 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 486709 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 486709 T1504 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 486710 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 486710 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 486711 T1504 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 486711 T1504 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 486712 T1504 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 486712 T1504 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 486713 T1504 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 486713 T1504 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54351/solr
   [junit4]   2> 486714 T1504 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 486716 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 486730 T1564 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cd26532 name:ZooKeeperConnection Watcher:127.0.0.1:54351 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 486731 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 486739 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 486744 T1566 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19aec824 name:ZooKeeperConnection Watcher:127.0.0.1:54351/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 486744 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 486771 T1504 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 487783 T1504 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54361_oyxj
   [junit4]   2> 487785 T1504 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54361_oyxj
   [junit4]   2> 487795 T1526 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 487796 T1535 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 487796 T1548 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 487797 T1566 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 487816 T1567 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 487816 T1567 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 487819 T1567 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 487819 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 487819 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 487820 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 487825 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 487827 T1527 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54361/oyxj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54361_oyxj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 487827 T1527 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 487828 T1527 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
   [junit4]   2> 487836 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 487838 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 487838 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 487839 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 487838 T1566 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 488820 T1567 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 488821 T1567 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 488824 T1567 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 488824 T1567 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 488827 T1567 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 488827 T1567 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004/collection1/'
   [junit4]   2> 488829 T1567 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 488830 T1567 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 488830 T1567 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 488907 T1567 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_4_9
   [junit4]   2> 488964 T1567 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 488967 T1567 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 488979 T1567 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 489404 T1567 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 489408 T1567 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 489408 T1567 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 489416 T1567 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 489482 T1567 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 489483 T1567 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 489483 T1567 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-004/collection1/, dataDir=./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty2/
   [junit4]   2> 489483 T1567 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2539efe5
   [junit4]   2> 489484 T1567 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty2
   [junit4]   2> 489485 T1567 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty2/index/
   [junit4]   2> 489486 T1567 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 489487 T1567 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty2/index
   [junit4]   2> 489488 T1567 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=16, maxMergedSegmentMB=79.1689453125, floorSegmentMB=0.3916015625, forceMergeDeletesPctAllowed=14.097871798244634, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 489494 T1567 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 489494 T1567 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 489502 T1567 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 489502 T1567 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 489503 T1567 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 489503 T1567 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 489504 T1567 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 489504 T1567 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 489505 T1567 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 489505 T1567 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 489505 T1567 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 489506 T1567 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 489507 T1567 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 489507 T1567 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 489507 T1567 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 489508 T1567 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 489509 T1567 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 489509 T1567 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 489522 T1567 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 489527 T1567 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 489527 T1567 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 489528 T1567 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=21, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=24.1015625, floorSegmentMB=0.21875, forceMergeDeletesPctAllowed=24.631939252862697, segmentsPerTier=47.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3203946024898354
   [junit4]   2> 489530 T1567 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 489530 T1567 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 489531 T1567 oass.SolrIndexSearcher.<init> Opening Searcher@51b69582[collection1] main
   [junit4]   2> 489531 T1567 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 489535 T1567 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 489535 T1567 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 489537 T1567 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 489538 T1567 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 489538 T1567 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 489540 T1567 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 489540 T1567 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 489541 T1567 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
   [junit4]   2> 489541 T1567 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 489550 T1568 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51b69582[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 489562 T1567 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 489565 T1571 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54361/oyxj collection:collection1 shard:shard3
   [junit4]   2> 489567 T1504 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 489567 T1504 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 489568 T1571 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
   [junit4]   2> 489581 T1571 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
   [junit4]   2> 489586 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 489589 T1571 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 489589 T1571 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C791 name=collection1 org.apache.solr.core.SolrCore@1c0540fa url=http://127.0.0.1:54361/oyxj/collection1 node=127.0.0.1:54361_oyxj C791_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:54361/oyxj, core=collection1, node_name=127.0.0.1:54361_oyxj}
   [junit4]   2> 489589 T1571 C791 P54361 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54361/oyxj/collection1/
   [junit4]   2> 489589 T1571 C791 P54361 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 489590 T1571 C791 P54361 oasc.SyncStrategy.syncToMe http://127.0.0.1:54361/oyxj/collection1/ has no replicas
   [junit4]   2> 489590 T1571 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54361/oyxj/collection1/ shard3
   [junit4]   2> 489590 T1571 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
   [junit4]   2> 489595 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 489610 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 489612 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489612 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489613 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489614 T1566 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489645 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 489655 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 489766 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489766 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489768 T1566 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489766 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489795 T1571 oasc.ZkController.register We are http://127.0.0.1:54361/oyxj/collection1/ and leader is http://127.0.0.1:54361/oyxj/collection1/
   [junit4]   2> 489795 T1571 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54361/oyxj
   [junit4]   2> 489796 T1571 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 489796 T1571 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 489797 T1571 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 489803 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 489803 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 489804 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 489807 T1571 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 489811 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 489814 T1527 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54361/oyxj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54361_oyxj",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 489828 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 489938 T1566 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489939 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489939 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 489939 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 491485 T1504 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 491486 T1504 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 491492 T1504 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54364
   [junit4]   2> 491494 T1504 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 491495 T1504 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 491495 T1504 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-005
   [junit4]   2> 491496 T1504 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-005/'
   [junit4]   2> 491543 T1504 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-005/solr.xml
   [junit4]   2> 491643 T1504 oasc.CoreContainer.<init> New CoreContainer 79927294
   [junit4]   2> 491643 T1504 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-005/]
   [junit4]   2> 491645 T1504 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 491645 T1504 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 491645 T1504 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 491646 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 491646 T1504 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 491646 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 491647 T1504 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 491647 T1504 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 491648 T1504 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 491648 T1504 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 491648 T1504 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 491649 T1504 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 491649 T1504 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54351/solr
   [junit4]   2> 491649 T1504 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 491651 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 491656 T1582 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@452fa723 name:ZooKeeperConnection Watcher:127.0.0.1:54351 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 491658 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 491664 T1504 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 491668 T1584 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2608c64b name:ZooKeeperConnection Watcher:127.0.0.1:54351/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 491669 T1504 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 491698 T1504 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 492711 T1504 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54364_oyxj
   [junit4]   2> 492714 T1504 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54364_oyxj
   [junit4]   2> 492726 T1526 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 492727 T1584 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 492727 T1548 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 492728 T1566 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 492728 T1535 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 492749 T1585 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 492749 T1585 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 492752 T1585 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 492752 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 492753 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 492753 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 492759 T1527 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 492761 T1527 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:54364/oyxj",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:54364_oyxj",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"3",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 492762 T1527 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
   [junit4]   2> 492762 T1527 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 492771 T1526 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 492774 T1526 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 492775 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 492775 T1566 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 492775 T1535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 492775 T1548 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 493754 T1585 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 493755 T1585 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 493757 T1585 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 493758 T1585 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 493762 T1585 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 493762 T1585 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-005/collection1/'
   [junit4]   2> 493765 T1585 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-005/collection1/lib/.svn/' to classloader
   [junit4]   2> 493766 T1585 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-005/collection1/lib/classes/' to classloader
   [junit4]   2> 493767 T1585 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/solr.cloud.ChaosMonkeySafeLeaderTest-D0D8F82F992F0231-001/tempDir-005/collection1/lib/README' to classloader
   [junit4]   2> 493896 T1585 oasc.SolrConfig.<init> Using Lucene MatchV

[...truncated too long message...]

     at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  71) Thread[id=1583, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[D0D8F82F992F0231]-SendThread(127.0.0.1:54351), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 	  72) Thread[id=1633, name=qtp1715234782-1633 Acceptor0 SelectChannelConnector@127.0.0.1:54377, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  73) Thread[id=1834, name=qtp862927534-1834 Acceptor0 SelectChannelConnector@127.0.0.1:54401, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  74) Thread[id=1621, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[D0D8F82F992F0231]-SendThread(127.0.0.1:54351), state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 	  75) Thread[id=1913, name=qtp1042943910-1913 Acceptor0 SelectChannelConnector@127.0.0.1:54383, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  76) Thread[id=1843, name=Thread-867-SendThread(127.0.0.1:54351), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]   2> 	        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]   2> 7711900 T1922 oaz.ClientCnxn$SendThread.run WARN Session 0x1466e6f60df0026 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   [junit4]   2> 	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=ar_LY, timezone=America/Noronha
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=2,threads=88,free=53741464,total=119881728
   [junit4]   2> NOTE: All tests run in this JVM: [TestSort, TestBinaryResponseWriter, TestReplicationHandlerBackup, CacheHeaderTest, TestLMDirichletSimilarityFactory, SpellingQueryConverterTest, HdfsCollectionsAPIDistributedZkTest, TestMultiCoreConfBootstrap, TestBlendedInfixSuggestions, OverseerStatusTest, TestFiltering, UUIDFieldTest, TestCloudManagedSchemaAddField, PeerSyncTest, ShardRoutingCustomTest, TestNumberUtils, TestConfigSets, TestAtomicUpdateErrorCases, TestCustomSort, StandardRequestHandlerTest, TestQueryTypes, ZkNodePropsTest, DistanceFunctionTest, TestSearchPerf, TestZkChroot, DeleteInactiveReplicaTest, SynonymTokenizerTest, TestElisionMultitermQuery, DisMaxRequestHandlerTest, PolyFieldTest, TestSchemaNameResource, DistributedSuggestComponentTest, SOLR749Test, TestStressRecovery, RAMDirectoryFactoryTest, TestExceedMaxTermLength, TestArbitraryIndexDir, LukeRequestHandlerTest, SuggesterFSTTest, TestDefaultSimilarityFactory, TestJmxIntegration, TestReload, TestPartialUpdateDeduplication, TestRecoveryHdfs, TestReversedWildcardFilterFactory, ScriptEngineTest, SolrCmdDistributorTest, TestPHPSerializedResponseWriter, TestFieldCollectionResource, TestDistributedMissingSort, TestXIncludeConfig, TestSolrXml, FastVectorHighlighterTest, SpellCheckComponentTest, TestRemoteStreaming, TestValueSourceCache, SimplePostToolTest, TestCoreContainer, OverseerCollectionProcessorTest, LegacyHTMLStripCharFilterTest, IndexSchemaRuntimeFieldTest, JSONWriterTest, TestFastOutputStream, IndexSchemaTest, SimpleFacetsTest, TestSolrIndexConfig, TestManagedSchema, TestSurroundQueryParser, FileBasedSpellCheckerTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=D0D8F82F992F0231 -Dtests.slow=true -Dtests.locale=ar_LY -Dtests.timezone=America/Noronha -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D0D8F82F992F0231]:0)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1876, name=RecoveryThread, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]    > Caused by: java.lang.AssertionError: liveDocs.count()=6 info.docCount=10 info.getDelCount()=5
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D0D8F82F992F0231]:0)
   [junit4]    > 	at org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:92)
   [junit4]    > 	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:110)
   [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:143)
   [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:237)
   [junit4]    > 	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:98)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:393)
   [junit4]    > 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:273)
   [junit4]    > 	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:251)
   [junit4]    > 	at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1471)
   [junit4]    > 	at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1672)
   [junit4]    > 	at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:673)
   [junit4]    > 	at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:493)
   [junit4]    > 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
   [junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
   [junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
   [junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
   [junit4] Completed in 7237.03s, 1 test, 3 errors <<< FAILURES!

[...truncated 987 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:467: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:447: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:45: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:496: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1297: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:921: There were test failures: 398 suites, 1642 tests, 2 suite-level errors, 1 error, 1336 ignored (8 assumptions)

Total time: 174 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any