You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/02/02 16:00:34 UTC

[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1287 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1287/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([62C4FAB36E0F11C8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([62C4FAB36E0F11C8]:0)




Build Log:
[...truncated 10886 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
   [junit4]   2> 2430963 T8263 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 2430970 T8263 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeyNothingIsSafeTest-1391345878346
   [junit4]   2> 2430971 T8263 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2430972 T8264 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2431073 T8263 oasc.ZkTestServer.run start zk server on port:51449
   [junit4]   2> 2431076 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2431090 T8270 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4dfff181 name:ZooKeeperConnection Watcher:127.0.0.1:51449 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2431091 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2431092 T8263 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2431103 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2431107 T8272 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6505e4a6 name:ZooKeeperConnection Watcher:127.0.0.1:51449/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2431108 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2431108 T8263 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2431117 T8263 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2431125 T8263 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2431133 T8263 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2431144 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2431145 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2431159 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 2431160 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2431174 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2431175 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2431193 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2431195 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2431204 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2431205 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2431212 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2431213 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2431223 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2431224 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2431233 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2431233 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2431245 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2431245 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2431253 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2431254 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2431264 T8263 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2431265 T8263 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2431615 T8263 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2431624 T8263 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51452
   [junit4]   2> 2431625 T8263 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2431626 T8263 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2431626 T8263 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651
   [junit4]   2> 2431626 T8263 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651/'
   [junit4]   2> 2431656 T8263 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651/solr.xml
   [junit4]   2> 2431729 T8263 oasc.CoreContainer.<init> New CoreContainer 1220649596
   [junit4]   2> 2431729 T8263 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651/]
   [junit4]   2> 2431731 T8263 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2431731 T8263 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2431731 T8263 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2431731 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2431732 T8263 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2431732 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2431733 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2431733 T8263 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2431733 T8263 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2431743 T8263 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2431744 T8263 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2431744 T8263 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2431744 T8263 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51449/solr
   [junit4]   2> 2431745 T8263 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2431746 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2431751 T8283 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2092a035 name:ZooKeeperConnection Watcher:127.0.0.1:51449 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2431751 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2431757 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2431761 T8285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ca29e17 name:ZooKeeperConnection Watcher:127.0.0.1:51449/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2431761 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2431766 T8263 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2431775 T8263 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2431783 T8263 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2431788 T8263 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51452_
   [junit4]   2> 2431790 T8263 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51452_
   [junit4]   2> 2431799 T8263 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2431805 T8263 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2431816 T8263 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:51452_
   [junit4]   2> 2431816 T8263 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2431821 T8263 oasc.Overseer.start Overseer (id=91183243483873283-127.0.0.1:51452_-n_0000000000) starting
   [junit4]   2> 2431830 T8263 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2431840 T8287 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2431841 T8263 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2431842 T8287 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2431847 T8263 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2431851 T8263 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2431858 T8286 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2431868 T8288 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2431868 T8288 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2431871 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2431871 T8288 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2431875 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2431877 T8286 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51452",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51452_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2431877 T8286 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 2431877 T8286 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2431885 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2431887 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2432872 T8288 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2432873 T8288 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651/collection1
   [junit4]   2> 2432874 T8288 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 2432876 T8288 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2432876 T8288 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 2432881 T8288 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651/collection1/'
   [junit4]   2> 2432882 T8288 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651/collection1/lib/classes/' to classloader
   [junit4]   2> 2432883 T8288 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651/collection1/lib/README' to classloader
   [junit4]   2> 2432927 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
   [junit4]   2> 2432963 T8288 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2433035 T8288 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2433040 T8288 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2433049 T8288 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2433791 T8288 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2433794 T8288 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2433798 T8288 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2433804 T8288 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2433856 T8288 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2433857 T8288 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391345878651/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/control/data/
   [junit4]   2> 2433857 T8288 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4620800
   [junit4]   2> 2433858 T8288 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/control/data
   [junit4]   2> 2433859 T8288 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/control/data/index/
   [junit4]   2> 2433860 T8288 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2433860 T8288 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/control/data/index
   [junit4]   2> 2433861 T8288 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=60.76171875, floorSegmentMB=1.953125, forceMergeDeletesPctAllowed=5.664416455061678, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2433866 T8288 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2433867 T8288 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2433872 T8288 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2433873 T8288 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2433874 T8288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2433874 T8288 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2433876 T8288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2433878 T8288 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2433884 T8288 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2433887 T8288 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2433887 T8288 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2433888 T8288 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2433888 T8288 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2433888 T8288 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2433889 T8288 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2433889 T8288 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2433890 T8288 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2433891 T8288 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2433891 T8288 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2433901 T8288 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2433906 T8288 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2433906 T8288 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2433908 T8288 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10852582241018745]
   [junit4]   2> 2433910 T8288 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/control/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 2433910 T8288 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2433910 T8288 oass.SolrIndexSearcher.<init> Opening Searcher@c348342 main
   [junit4]   2> 2433914 T8289 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c348342 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2433920 T8288 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2433925 T8263 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2433925 T8263 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2433925 T8292 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51452 collection:control_collection shard:shard1
   [junit4]   2> 2433928 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2433933 T8294 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a81f937 name:ZooKeeperConnection Watcher:127.0.0.1:51449/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2433933 T8292 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2433934 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2433936 T8263 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2433943 T8263 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2433976 T8292 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2433984 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2433986 T8292 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2433986 T8292 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C6130 name=collection1 org.apache.solr.core.SolrCore@2ab697f url=http://127.0.0.1:51452/collection1 node=127.0.0.1:51452_ C6130_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:51452, core=collection1, node_name=127.0.0.1:51452_}
   [junit4]   2> 2433987 T8292 C6130 P51452 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51452/collection1/
   [junit4]   2> 2433987 T8292 C6130 P51452 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2433988 T8292 C6130 P51452 oasc.SyncStrategy.syncToMe http://127.0.0.1:51452/collection1/ has no replicas
   [junit4]   2> 2433988 T8292 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51452/collection1/ shard1
   [junit4]   2> 2433989 T8292 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2433995 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2434015 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2434019 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2434021 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2434044 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2434054 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2434162 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2434162 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2434185 T8292 oasc.ZkController.register We are http://127.0.0.1:51452/collection1/ and leader is http://127.0.0.1:51452/collection1/
   [junit4]   2> 2434186 T8292 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51452
   [junit4]   2> 2434186 T8292 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2434186 T8292 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2434187 T8292 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2434190 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2434190 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2434190 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2434193 T8292 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2434204 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2434209 T8286 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51452",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51452_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2434220 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2434327 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2434328 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 2434386 T8263 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2434388 T8263 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2434395 T8263 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51456
   [junit4]   2> 2434395 T8263 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2434395 T8263 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2434396 T8263 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320
   [junit4]   2> 2434397 T8263 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320/'
   [junit4]   2> 2434467 T8263 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320/solr.xml
   [junit4]   2> 2434564 T8263 oasc.CoreContainer.<init> New CoreContainer 44133770
   [junit4]   2> 2434565 T8263 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320/]
   [junit4]   2> 2434567 T8263 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2434567 T8263 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2434567 T8263 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2434567 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2434568 T8263 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2434568 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2434568 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2434569 T8263 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2434569 T8263 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2434582 T8263 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2434585 T8263 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2434585 T8263 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2434585 T8263 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51449/solr
   [junit4]   2> 2434586 T8263 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2434587 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2434592 T8305 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@341e805f name:ZooKeeperConnection Watcher:127.0.0.1:51449 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2434593 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2434597 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2434602 T8307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b237625 name:ZooKeeperConnection Watcher:127.0.0.1:51449/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2434602 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2434611 T8263 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2435638 T8263 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51456_
   [junit4]   2> 2435644 T8263 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51456_
   [junit4]   2> 2435657 T8285 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2435657 T8294 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2435658 T8307 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2435694 T8308 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2435695 T8308 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2435699 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2435700 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2435700 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2435699 T8308 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2435709 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2435712 T8286 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51456",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51456_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2435713 T8286 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 2435713 T8286 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2435727 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2435838 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2435839 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2435838 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2436703 T8308 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2436704 T8308 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320/collection1
   [junit4]   2> 2436704 T8308 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2436707 T8308 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2436708 T8308 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2436717 T8308 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320/collection1/'
   [junit4]   2> 2436719 T8308 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320/collection1/lib/classes/' to classloader
   [junit4]   2> 2436720 T8308 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320/collection1/lib/README' to classloader
   [junit4]   2> 2436823 T8308 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2436917 T8308 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2436926 T8308 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2436940 T8308 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2437584 T8308 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2437587 T8308 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2437590 T8308 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2437597 T8308 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2437672 T8308 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2437672 T8308 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391345881320/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty1/
   [junit4]   2> 2437673 T8308 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4620800
   [junit4]   2> 2437674 T8308 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty1
   [junit4]   2> 2437674 T8308 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty1/index/
   [junit4]   2> 2437675 T8308 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 2437675 T8308 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty1/index
   [junit4]   2> 2437676 T8308 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=60.76171875, floorSegmentMB=1.953125, forceMergeDeletesPctAllowed=5.664416455061678, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2437679 T8308 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2437679 T8308 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2437685 T8308 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2437686 T8308 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2437686 T8308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2437687 T8308 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2437687 T8308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2437687 T8308 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2437688 T8308 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2437689 T8308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2437689 T8308 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2437690 T8308 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2437691 T8308 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2437692 T8308 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2437692 T8308 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2437694 T8308 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2437695 T8308 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2437696 T8308 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2437697 T8308 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2437707 T8308 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2437711 T8308 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2437711 T8308 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2437712 T8308 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10852582241018745]
   [junit4]   2> 2437713 T8308 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty1/index,segFN=segments_1,generation=1}
   [junit4]   2> 2437713 T8308 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2437714 T8308 oass.SolrIndexSearcher.<init> Opening Searcher@1aa2d79b main
   [junit4]   2> 2437725 T8309 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1aa2d79b main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2437729 T8308 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2437731 T8263 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2437732 T8312 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51456 collection:collection1 shard:shard2
   [junit4]   2> 2437732 T8263 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2437734 T8312 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 2437746 T8312 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 2437751 T8312 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2437751 T8312 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C6131 name=collection1 org.apache.solr.core.SolrCore@2174ce73 url=http://127.0.0.1:51456/collection1 node=127.0.0.1:51456_ C6131_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51456, core=collection1, node_name=127.0.0.1:51456_}
   [junit4]   2> 2437751 T8312 C6131 P51456 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51456/collection1/
   [junit4]   2> 2437752 T8312 C6131 P51456 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2437752 T8312 C6131 P51456 oasc.SyncStrategy.syncToMe http://127.0.0.1:51456/collection1/ has no replicas
   [junit4]   2> 2437753 T8312 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51456/collection1/ shard2
   [junit4]   2> 2437753 T8312 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 2437753 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2437754 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2437754 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2437762 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2437769 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2437773 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2437776 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2437777 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2437783 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2437790 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2437900 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2437900 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2437901 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2437924 T8312 oasc.ZkController.register We are http://127.0.0.1:51456/collection1/ and leader is http://127.0.0.1:51456/collection1/
   [junit4]   2> 2437925 T8312 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51456
   [junit4]   2> 2437925 T8312 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2437925 T8312 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2437925 T8312 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2437927 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2437928 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2437928 T8312 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2437928 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2437933 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2437934 T8286 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51456",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51456_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node1"}
   [junit4]   2> 2437948 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2438054 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2438054 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2438054 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 2438175 T8263 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2438177 T8263 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2438185 T8263 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51459
   [junit4]   2> 2438185 T8263 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2438185 T8263 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2438186 T8263 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109
   [junit4]   2> 2438186 T8263 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109/'
   [junit4]   2> 2438221 T8263 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109/solr.xml
   [junit4]   2> 2438311 T8263 oasc.CoreContainer.<init> New CoreContainer 1492042889
   [junit4]   2> 2438311 T8263 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109/]
   [junit4]   2> 2438313 T8263 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2438313 T8263 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2438313 T8263 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2438314 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2438314 T8263 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2438314 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2438315 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2438315 T8263 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2438315 T8263 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2438325 T8263 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2438325 T8263 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2438325 T8263 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2438326 T8263 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51449/solr
   [junit4]   2> 2438326 T8263 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2438327 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2438334 T8323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23a92bfc name:ZooKeeperConnection Watcher:127.0.0.1:51449 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2438334 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2438340 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2438345 T8325 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d06f906 name:ZooKeeperConnection Watcher:127.0.0.1:51449/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2438346 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2438357 T8263 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2439366 T8263 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51459_
   [junit4]   2> 2439369 T8263 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51459_
   [junit4]   2> 2439379 T8325 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2439379 T8285 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2439379 T8294 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2439380 T8307 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2439406 T8326 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2439407 T8326 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2439410 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2439410 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2439411 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2439410 T8326 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2439415 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2439417 T8286 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51459",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51459_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2439418 T8286 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 2439418 T8286 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 2439437 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2439545 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2439546 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2439547 T8325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2439547 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2440417 T8326 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2440418 T8326 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109/collection1
   [junit4]   2> 2440418 T8326 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2440421 T8326 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2440421 T8326 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2440425 T8326 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109/collection1/'
   [junit4]   2> 2440427 T8326 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109/collection1/lib/classes/' to classloader
   [junit4]   2> 2440428 T8326 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109/collection1/lib/README' to classloader
   [junit4]   2> 2440512 T8326 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2440614 T8326 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2440617 T8326 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2440632 T8326 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2441173 T8326 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2441175 T8326 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2441178 T8326 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2441184 T8326 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2441243 T8326 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2441244 T8326 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391345885109/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty2/
   [junit4]   2> 2441244 T8326 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4620800
   [junit4]   2> 2441245 T8326 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty2
   [junit4]   2> 2441246 T8326 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty2/index/
   [junit4]   2> 2441246 T8326 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 2441247 T8326 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty2/index
   [junit4]   2> 2441247 T8326 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=60.76171875, floorSegmentMB=1.953125, forceMergeDeletesPctAllowed=5.664416455061678, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2441250 T8326 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2441251 T8326 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2441255 T8326 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2441256 T8326 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2441256 T8326 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2441257 T8326 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2441257 T8326 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2441258 T8326 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2441258 T8326 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2441259 T8326 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2441259 T8326 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2441260 T8326 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2441261 T8326 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2441262 T8326 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2441262 T8326 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2441263 T8326 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2441264 T8326 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2441264 T8326 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2441265 T8326 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2441276 T8326 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2441280 T8326 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2441280 T8326 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2441282 T8326 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.10852582241018745]
   [junit4]   2> 2441284 T8326 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty2/index,segFN=segments_1,generation=1}
   [junit4]   2> 2441284 T8326 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2441285 T8326 oass.SolrIndexSearcher.<init> Opening Searcher@6544fabc main
   [junit4]   2> 2441291 T8327 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6544fabc main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2441295 T8326 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 2441297 T8263 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2441297 T8263 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2441334 T8330 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51459 collection:collection1 shard:shard1
   [junit4]   2> 2441393 T8330 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 2441426 T8330 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 2441430 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441431 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441431 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441433 T8330 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 2441433 T8330 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C6132 name=collection1 org.apache.solr.core.SolrCore@5200b519 url=http://127.0.0.1:51459/collection1 node=127.0.0.1:51459_ C6132_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:51459, core=collection1, node_name=127.0.0.1:51459_}
   [junit4]   2> 2441433 T8330 C6132 P51459 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51459/collection1/
   [junit4]   2> 2441433 T8330 C6132 P51459 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 2441434 T8330 C6132 P51459 oasc.SyncStrategy.syncToMe http://127.0.0.1:51459/collection1/ has no replicas
   [junit4]   2> 2441434 T8330 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:51459/collection1/ shard1
   [junit4]   2> 2441434 T8330 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 2441436 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2441447 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441450 T8325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441450 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441450 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441451 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441469 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2441476 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441582 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441582 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441582 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441583 T8325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441607 T8330 oasc.ZkController.register We are http://127.0.0.1:51459/collection1/ and leader is http://127.0.0.1:51459/collection1/
   [junit4]   2> 2441608 T8330 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51459
   [junit4]   2> 2441608 T8330 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 2441608 T8330 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 2441609 T8330 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2441611 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441611 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441611 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441612 T8330 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2441616 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2441617 T8286 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51459",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51459_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":"core_node2"}
   [junit4]   2> 2441624 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2441724 T8263 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2441725 T8263 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2441730 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441731 T8325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441731 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441731 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 2441737 T8263 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51462
   [junit4]   2> 2441737 T8263 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2441737 T8263 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2441738 T8263 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673
   [junit4]   2> 2441738 T8263 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673/'
   [junit4]   2> 2441768 T8263 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673/solr.xml
   [junit4]   2> 2441885 T8263 oasc.CoreContainer.<init> New CoreContainer 867512596
   [junit4]   2> 2441886 T8263 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673/]
   [junit4]   2> 2441888 T8263 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2441889 T8263 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2441889 T8263 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2441890 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2441890 T8263 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2441890 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2441891 T8263 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2441891 T8263 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2441891 T8263 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2441901 T8263 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2441902 T8263 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2441902 T8263 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2441903 T8263 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51449/solr
   [junit4]   2> 2441904 T8263 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2441905 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2441909 T8341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3ca93291 name:ZooKeeperConnection Watcher:127.0.0.1:51449 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2441910 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2441916 T8263 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2441919 T8343 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d8a7e48 name:ZooKeeperConnection Watcher:127.0.0.1:51449/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2441920 T8263 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2441929 T8263 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2442941 T8263 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51462_
   [junit4]   2> 2442946 T8263 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51462_
   [junit4]   2> 2442963 T8294 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2442967 T8285 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2442968 T8343 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2442968 T8307 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2442968 T8325 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2442998 T8344 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 2442998 T8344 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 2443003 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2443003 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2443004 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2443003 T8344 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2443010 T8286 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2443013 T8286 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51462",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:51462_",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2443014 T8286 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 2443014 T8286 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 2443029 T8285 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2443139 T8285 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2443139 T8343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2443139 T8307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2443140 T8294 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2443139 T8325 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 2444009 T8344 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2444009 T8344 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673/collection1
   [junit4]   2> 2444009 T8344 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 2444011 T8344 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2444011 T8344 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 2444014 T8344 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673/collection1/'
   [junit4]   2> 2444016 T8344 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673/collection1/lib/classes/' to classloader
   [junit4]   2> 2444017 T8344 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673/collection1/lib/README' to classloader
   [junit4]   2> 2444133 T8344 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2444232 T8344 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2444236 T8344 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2444246 T8344 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 2445073 T8344 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2445078 T8344 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2445081 T8344 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2445092 T8344 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2445184 T8344 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
   [junit4]   2> 2445184 T8344 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391345888673/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty3/
   [junit4]   2> 2445185 T8344 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4620800
   [junit4]   2> 2445187 T8344 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty3
   [junit4]   2> 2445188 T8344 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty3/index/
   [junit4]   2> 2445189 T8344 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 2445190 T8344 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty3/index
   [junit4]   2> 2445191 T8344 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=19, maxMergedSegmentMB=60.76171875, floorSegmentMB=1.953125, forceMergeDeletesPctAllowed=5.664416455061678, segmentsPerTier=37.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 2445208 T8344 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391345878346/jetty3/index,segFN=segments_1,generation=1}
   [junit4]   2> 2445209 T8344 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2445223 T8344 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2445224 T8344 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2445225 T8344 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2445226 T8344 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2445226 T8344 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 244522

[...truncated too long message...]

e(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> 	  36) Thread[id=8388, name=coreZkRegister-5796-thread-1, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
   [junit4]   2> 	        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:744)
   [junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=cs, timezone=America/St_Vincent
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=43,free=337585328,total=438407168
   [junit4]   2> NOTE: All tests run in this JVM: [InfoHandlerTest, TestStressReorder, TestZkChroot, TestRTGBase, TestStressVersions, SyncSliceTest, DeleteReplicaTest, ShardRoutingCustomTest, TestCollationFieldDocValues, SuggestComponentTest, DistributedDebugComponentTest, CopyFieldTest, CoreAdminHandlerTest, TestBM25SimilarityFactory, HdfsRecoveryZkTest, TestSolrQueryParser, DisMaxRequestHandlerTest, TestCloudManagedSchemaAddField, AddBlockUpdateTest, TestFreeTextSuggestions, TestNoOpRegenerator, TestUpdate, LukeRequestHandlerTest, AnalysisAfterCoreReloadTest, TestUniqueKeyFieldResource, AddSchemaFieldsUpdateProcessorFactoryTest, BadCopyFieldTest, TestAnalyzedSuggestions, TestJmxIntegration, TestWriterPerf, ZkCLITest, OutputWriterTest, TermVectorComponentDistributedTest, TestWordDelimiterFilterFactory, TestAnalyzeInfixSuggestions, HardAutoCommitTest, SuggesterWFSTTest, ScriptEngineTest, TestDocSet, StatsComponentTest, TestAtomicUpdateErrorCases, AliasIntegrationTest, TestRemoteStreaming, TestPseudoReturnFields, HdfsBasicDistributedZkTest, TestDefaultSearchFieldResource, TestLazyCores, CursorPagingTest, ZkNodePropsTest, TestStressRecovery, BadComponentTest, RangeFacetTest, TestPartialUpdateDeduplication, TestReload, DocValuesMultiTest, TestCSVLoader, TestFieldTypeResource, TestManagedSchemaFieldResource, CurrencyFieldOpenExchangeTest, LegacyHTMLStripCharFilterTest, TestXIncludeConfig, TestSearchPerf, PolyFieldTest, HdfsLockFactoryTest, TestRealTimeGet, TermsComponentTest, StressHdfsTest, TestSolrXml, SolrTestCaseJ4Test, HighlighterConfigTest, SpellingQueryConverterTest, TestCoreDiscovery, DefaultValueUpdateProcessorTest, FunctionTest, CoreContainerCoreInitFailuresTest, TestJoin, TestSchemaResource, TestSolrDeletionPolicy2, CursorMarkTest, TestConfig, TestDistributedGrouping, SuggesterTest, TestCharFilters, TestFieldResource, SolrIndexSplitterTest, TestDocBasedVersionConstraints, DistributedQueryElevationComponentTest, TestIndexingPerformance, TestSurroundQueryParser, HdfsSyncSliceTest, XsltUpdateRequestHandlerTest, AlternateDirectoryTest, SynonymTokenizerTest, TestFaceting, DistributedSuggestComponentTest, MultiTermTest, NotRequiredUniqueKeyTest, TestFunctionQuery, FullSolrCloudDistribCmdsTest, RAMDirectoryFactoryTest, LoggingHandlerTest, ClusterStateTest, TestFileDictionaryLookup, MoreLikeThisHandlerTest, TestPerFieldSimilarity, JsonLoaderTest, TestIBSimilarityFactory, ResourceLoaderTest, TestQuerySenderListener, FieldAnalysisRequestHandlerTest, OverseerTest, RecoveryZkTest, DocumentAnalysisRequestHandlerTest, TestReloadAndDeleteDocs, SearchHandlerTest, TestArbitraryIndexDir, PingRequestHandlerTest, ConvertedLegacyTest, BlockDirectoryTest, TestSort, TestSimpleQParserPlugin, ClusterStateUpdateTest, TestSolrXmlPersistence, TestCodecSupport, TestBadConfig, TestRangeQuery, ShardRoutingTest, TestFiltering, DirectUpdateHandlerOptimizeTest, CurrencyFieldXmlFileTest, JSONWriterTest, TestBinaryField, TestLRUCache, BJQParserTest, TestRandomFaceting, TestHighFrequencyDictionaryFactory, TestSolrXmlPersistor, QueryFacetTest, TestNumberUtils, TestHashPartitioner, DistributedTermsComponentTest, TestLMJelinekMercerSimilarityFactory, HdfsCollectionsAPIDistributedZkTest, PathHierarchyTokenizerFactoryTest, TestQueryTypes, TestSweetSpotSimilarityFactory, BasicDistributedZk2Test, PreAnalyzedFieldTest, SpellPossibilityIteratorTest, QueryParsingTest, TestSchemaVersionResource, PeerSyncTest, TestRecovery, UpdateParamsTest, ReturnFieldsTest, FastVectorHighlighterTest, TestDistributedSearch, CacheHeaderTest, HdfsDirectoryTest, TestOmitPositions, AssignTest, FileUtilsTest, SolrXmlInZkTest, DirectSolrConnectionTest, TestStressUserVersions, TestQuerySenderNoQuery, TestFieldCollectionResource, OpenExchangeRatesOrgProviderTest, QueryResultKeyTest, TestDynamicFieldResource, CollectionsAPIDistributedZkTest, RequiredFieldsTest, SolrCoreTest, SOLR749Test, TestQueryUtils, SystemInfoHandlerTest, ShowFileRequestHandlerTest, HighlighterTest, QueryElevationComponentTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, BasicZkTest, TestReplicationHandler, LeaderElectionTest, ZkSolrClientTest, TestRandomDVFaceting, SliceStateUpdateTest, AutoCommitTest, DistributedSpellCheckComponentTest, ZkControllerTest, TestMultiCoreConfBootstrap, TestCoreContainer, SimpleFacetsTest, TestSolr4Spatial, SpellCheckComponentTest, TestGroupingSearch, SolrCmdDistributorTest, BadIndexSchemaTest, BasicFunctionalityTest, DirectUpdateHandlerTest, OverseerCollectionProcessorTest, TestIndexSearcher, SoftAutoCommitTest, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SpellCheckCollatorTest, SuggesterFSTTest, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTSTTest, TestStressLucene, TestTrie, SpatialFilterTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, QueryEqualityTest, StatelessScriptUpdateProcessorFactoryTest, SortByFunctionTest, DistanceFunctionTest, TestSolrDeletionPolicy1, SolrInfoMBeanTest, DebugComponentTest, IndexBasedSpellCheckerTest, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, DocumentBuilderTest, TestValueSourceCache, TermVectorComponentTest, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, SolrPluginUtilsTest, IndexSchemaTest, TestCollationField, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, MBeansHandlerTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, TestComponentsName, TestBinaryResponseWriter, TestLFUCache, TestSolrIndexConfig, ResponseLogComponentTest, SolrIndexConfigTest, TestMergePolicyConfig, SampleTest, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestSolrCoreProperties, TestPostingsSolrHighlighter, TestLuceneMatchVersion, TestPhraseSuggestions, EchoParamsTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestDefaultSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, TimeZoneUtilsTest, TestFastOutputStream, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.seed=62C4FAB36E0F11C8 -Dtests.slow=true -Dtests.locale=cs -Dtests.timezone=America/St_Vincent -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s | ChaosMonkeyNothingIsSafeTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([62C4FAB36E0F11C8]:0)
   [junit4] Completed in 7266.31s, 1 test, 2 errors <<< FAILURES!

[...truncated 244 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: There were test failures: 365 suites, 1590 tests, 1 suite-level error, 1 error, 266 ignored (6 assumptions)

Total time: 193 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1289 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1289/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=1228, name=Overseer-91186736294920206-127.0.0.1:49523_-n_0000000005, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1228, name=Overseer-91186736294920206-127.0.0.1:49523_-n_0000000005, state=RUNNABLE, group=Overseer collection creation process.]
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([CF65042CE516355A]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:202)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qn-0000000004
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:273)
	at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:270)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:270)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:192)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:193)
	... 1 more




Build Log:
[...truncated 9915 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 439352 T1072 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 439359 T1072 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1391399174410
   [junit4]   2> 439360 T1072 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 439361 T1073 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 439365 T1072 oasc.ZkTestServer.run start zk server on port:49504
   [junit4]   2> 439384 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 439396 T1079 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7079e04c name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 439396 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 439397 T1072 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 439410 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 439412 T1081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a9fea98 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 439413 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 439413 T1072 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 439423 T1072 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 439431 T1072 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 439440 T1072 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 439448 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 439449 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 439461 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 439462 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 439473 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 439473 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 439483 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 439484 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 439494 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 439495 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 439504 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 439504 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 439523 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 439524 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 439535 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 439536 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 439551 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 439552 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 439562 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 439563 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 439571 T1072 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 439571 T1072 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 439984 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 439993 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49507
   [junit4]   2> 439993 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 439994 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 439994 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632
   [junit4]   2> 439994 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/'
   [junit4]   2> 440021 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/solr.xml
   [junit4]   2> 440098 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 440107 T1072 oasc.CoreContainer.<init> New CoreContainer 527817310
   [junit4]   2> 440107 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/]
   [junit4]   2> 440108 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 440108 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 440109 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 440109 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 440109 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 440109 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 440110 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 440110 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 440110 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 440121 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 440121 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 440121 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 440122 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 440122 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 440123 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440129 T1092 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72536174 name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440131 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440140 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440145 T1094 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32de1a32 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440146 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440150 T1072 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 440161 T1072 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 440170 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 440175 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49507_
   [junit4]   2> 440178 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49507_
   [junit4]   2> 440188 T1072 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 440196 T1072 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 440213 T1072 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:49507_
   [junit4]   2> 440213 T1072 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 440224 T1072 oasc.Overseer.start Overseer (id=91186736294920195-127.0.0.1:49507_-n_0000000000) starting
   [junit4]   2> 440238 T1072 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 440260 T1096 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 440265 T1072 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 440265 T1096 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 440274 T1072 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 440280 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 440292 T1095 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 440305 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632
   [junit4]   2> 440411 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/conf/
   [junit4]   2> 440416 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 440418 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 440418 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 440420 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440425 T1099 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d655ccd name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440426 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440431 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 440439 T1072 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 440798 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 440799 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 440803 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49511
   [junit4]   2> 440804 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 440805 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 440805 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490
   [junit4]   2> 440805 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/'
   [junit4]   2> 440837 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/solr.xml
   [junit4]   2> 440918 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 440919 T1072 oasc.CoreContainer.<init> New CoreContainer 279886935
   [junit4]   2> 440919 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/]
   [junit4]   2> 440921 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 440921 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 440922 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 440922 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 440922 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 440923 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 440923 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 440923 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 440923 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 440935 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 440936 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 440936 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 440937 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 440937 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 440939 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440943 T1110 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fcefc4f name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440944 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440950 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 440955 T1112 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6844f2eb name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 440955 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 440968 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 441978 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49511_
   [junit4]   2> 441981 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49511_
   [junit4]   2> 441992 T1112 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 441992 T1099 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 441996 T1094 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 442006 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490
   [junit4]   2> 442120 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/conf/
   [junit4]   2> 442127 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 442127 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 442127 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 442471 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 442472 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 442476 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49514
   [junit4]   2> 442478 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 442478 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 442479 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179
   [junit4]   2> 442479 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/'
   [junit4]   2> 442509 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/solr.xml
   [junit4]   2> 442588 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 442588 T1072 oasc.CoreContainer.<init> New CoreContainer 141531096
   [junit4]   2> 442589 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/]
   [junit4]   2> 442590 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 442590 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 442591 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 442591 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 442591 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 442591 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 442592 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 442592 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 442593 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 442603 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 442604 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 442604 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 442604 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 442604 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 442606 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 442610 T1124 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5659ea3b name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 442611 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 442621 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 442625 T1126 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35a5c518 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 442626 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 442640 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 443652 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49514_
   [junit4]   2> 443655 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49514_
   [junit4]   2> 443664 T1126 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 443665 T1112 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 443665 T1094 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 443666 T1099 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 443683 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179
   [junit4]   2> 443795 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/conf/
   [junit4]   2> 443802 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 443803 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 443803 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 444156 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 444158 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 444162 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49517
   [junit4]   2> 444164 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 444164 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 444164 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856
   [junit4]   2> 444165 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/'
   [junit4]   2> 444198 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/solr.xml
   [junit4]   2> 444276 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 444277 T1072 oasc.CoreContainer.<init> New CoreContainer 1017154163
   [junit4]   2> 444277 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/]
   [junit4]   2> 444278 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 444278 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 444279 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 444279 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 444279 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 444280 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 444280 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 444280 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 444280 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 444292 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 444293 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 444293 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 444294 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 444294 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 444295 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444299 T1138 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fe1a31d name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444300 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444307 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 444311 T1140 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16f28bac name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 444312 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 444326 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 445340 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49517_
   [junit4]   2> 445345 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49517_
   [junit4]   2> 445366 T1112 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445368 T1094 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445370 T1099 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445370 T1126 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445372 T1140 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 445408 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856
   [junit4]   2> 445515 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/conf/
   [junit4]   2> 445522 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 445523 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 445524 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 445958 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 445960 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 445969 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49520
   [junit4]   2> 445972 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 445973 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 445990 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576
   [junit4]   2> 445990 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/'
   [junit4]   2> 446023 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/solr.xml
   [junit4]   2> 446092 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 446093 T1072 oasc.CoreContainer.<init> New CoreContainer 509384364
   [junit4]   2> 446093 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/]
   [junit4]   2> 446095 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 446096 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 446096 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 446097 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 446097 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 446098 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 446098 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 446098 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 446098 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 446110 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 446111 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 446112 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 446112 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 446113 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 446114 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 446118 T1152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50a805c6 name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 446120 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446127 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 446132 T1154 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b238cce name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 446133 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 446146 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 447162 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49520_
   [junit4]   2> 447166 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49520_
   [junit4]   2> 447181 T1099 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447182 T1094 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447183 T1126 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447183 T1140 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447183 T1112 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447184 T1154 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 447198 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576
   [junit4]   2> 447365 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/conf/
   [junit4]   2> 447375 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 447376 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 447376 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 447749 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 447749 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 447756 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49523
   [junit4]   2> 447758 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 447758 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 447758 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428
   [junit4]   2> 447759 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/'
   [junit4]   2> 447788 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/solr.xml
   [junit4]   2> 447860 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 447861 T1072 oasc.CoreContainer.<init> New CoreContainer 1780100759
   [junit4]   2> 447861 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/]
   [junit4]   2> 447862 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 447863 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 447863 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 447863 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 447864 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 447864 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 447864 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 447864 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 447865 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 447877 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 447878 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 447878 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 447878 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 447879 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 447880 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447886 T1166 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ce8f1ec name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447887 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447895 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 447898 T1168 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1be0127b name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 447898 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 447912 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 448923 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49523_
   [junit4]   2> 448926 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49523_
   [junit4]   2> 448937 T1099 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448938 T1094 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448938 T1154 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448938 T1168 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448939 T1112 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448939 T1126 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448939 T1140 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 448957 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428
   [junit4]   2> 449117 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/conf/
   [junit4]   2> 449125 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 449126 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 449127 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 449484 T1072 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 449485 T1072 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 449489 T1072 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49526
   [junit4]   2> 449491 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 449492 T1072 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 449492 T1072 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178
   [junit4]   2> 449493 T1072 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/'
   [junit4]   2> 449525 T1072 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/solr.xml
   [junit4]   2> 449600 T1072 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 449600 T1072 oasc.CoreContainer.<init> New CoreContainer 1860417090
   [junit4]   2> 449601 T1072 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/]
   [junit4]   2> 449602 T1072 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 449602 T1072 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 449602 T1072 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 449603 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 449603 T1072 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 449603 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 449603 T1072 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 449604 T1072 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 449604 T1072 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 449620 T1072 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 449620 T1072 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 449621 T1072 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 449621 T1072 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:49504/solr
   [junit4]   2> 449621 T1072 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 449623 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449628 T1180 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@27d2dfee name:ZooKeeperConnection Watcher:127.0.0.1:49504 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449629 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449636 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 449641 T1182 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@35e406c1 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 449642 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 449658 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 450674 T1072 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49526_
   [junit4]   2> 450679 T1072 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49526_
   [junit4]   2> 450695 T1099 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450696 T1140 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1154 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1126 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1182 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1094 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1112 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450697 T1168 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 450715 T1072 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178
   [junit4]   2> 450877 T1072 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/conf/
   [junit4]   2> 450884 T1072 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 450885 T1072 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 450885 T1072 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 450887 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451594 T1185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2041a823 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451596 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451605 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 451620 T1072 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 451627 T1187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20ea3120 name:ZooKeeperConnection Watcher:127.0.0.1:49504/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 451628 T1072 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 451632 T1072 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 451647 T1089 oasha.CollectionsHandler.handleCreateAction Creating Collection : wt=javabin&replicationFactor=2&maxShardsPerNode=2&name=testOverseerCol&version=2&numShards=4&action=CREATE
   [junit4]   2> 451650 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451654 T1096 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 451654 T1096 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 451656 T1096 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 451656 T1096 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 451657 T1096 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 451668 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451673 T1095 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 451675 T1095 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 451675 T1095 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 451681 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451685 T1168 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1126 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451686 T1182 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451686 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451686 T1187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451686 T1154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451685 T1099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 451771 T1096 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 451771 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:49514_
   [junit4]   2> 451772 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:49526_
   [junit4]   2> 451774 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:49520_
   [junit4]   2> 451775 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:49517_
   [junit4]   2> 451778 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:49523_
   [junit4]   2> 451779 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:49511_
   [junit4]   2> 451782 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:49507_
   [junit4]   2> 451786 T1122 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 451787 T1096 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:49514_
   [junit4]   2> 451788 T1136 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 451793 T1147 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 451796 T1178 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 451797 T1157 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 451797 T1117 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 451799 T1122 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451803 T1090 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 451801 T1106 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 451804 T1136 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451801 T1147 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451801 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451805 T1117 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451805 T1178 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451807 T1106 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451803 T1157 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451807 T1090 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 451811 T1095 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 451813 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49514",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49514_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451820 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451823 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49520",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49520_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451831 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451834 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49523",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49523_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451845 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451849 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49517",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49517_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451856 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451859 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49526",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49526_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451867 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451871 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49514",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49514_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451879 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451882 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49511",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49511_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451890 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 451898 T1095 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:49507",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:49507_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 451906 T1094 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 452015 T1112 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452015 T1168 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452015 T1094 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452016 T1154 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452016 T1185 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452017 T1182 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452017 T1187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452017 T1126 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452016 T1099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452016 T1140 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 452804 T1122 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 452806 T1136 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 452808 T1147 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 452810 T1136 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/testOverseerCol_shard2_replica2
   [junit4]   2> 452810 T1136 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452811 T1117 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 452812 T1122 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1
   [junit4]   2> 452815 T1122 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452817 T1117 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard4_replica2
   [junit4]   2> 452819 T1117 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452812 T1106 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 452823 T1122 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452823 T1122 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452819 T1136 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452825 T1136 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452815 T1090 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 452827 T1136 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391399178856/testOverseerCol_shard2_replica2/'
   [junit4]   2> 452828 T1090 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/testOverseerCol_shard4_replica1
   [junit4]   2> 452829 T1090 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452830 T1122 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/'
   [junit4]   2> 452814 T1157 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 452813 T1178 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 452833 T1157 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1
   [junit4]   2> 452835 T1157 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452835 T1178 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/testOverseerCol_shard1_replica2
   [junit4]   2> 452843 T1178 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452813 T1147 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/testOverseerCol_shard2_replica1
   [junit4]   2> 452844 T1147 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452824 T1117 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452848 T1117 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452849 T1090 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452849 T1090 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452850 T1147 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452850 T1147 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452853 T1117 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard4_replica2/'
   [junit4]   2> 452824 T1106 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2
   [junit4]   2> 452857 T1106 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 452857 T1178 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452858 T1178 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452858 T1157 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452859 T1157 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452861 T1147 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391399180576/testOverseerCol_shard2_replica1/'
   [junit4]   2> 452878 T1106 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 452879 T1106 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 452892 T1090 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391399174632/testOverseerCol_shard4_replica1/'
   [junit4]   2> 452906 T1157 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/'
   [junit4]   2> 452917 T1178 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/testOverseerCol_shard1_replica2/'
   [junit4]   2> 452938 T1106 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2/'
   [junit4]   2> 453083 T1117 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453089 T1178 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453103 T1106 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453135 T1147 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453187 T1157 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453191 T1136 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453210 T1090 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453322 T1122 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 453368 T1117 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453372 T1117 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453380 T1106 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453390 T1117 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 453393 T1147 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453399 T1178 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453400 T1106 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453412 T1106 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 453418 T1178 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453437 T1178 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 453442 T1147 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453453 T1147 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 453502 T1136 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453512 T1136 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453521 T1136 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 453582 T1090 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453585 T1090 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453601 T1122 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453606 T1122 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453615 T1122 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 453625 T1090 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 453683 T1157 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 453688 T1157 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 453700 T1157 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 455726 T1122 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 455758 T1122 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 455784 T1122 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 455862 T1122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 455870 T1122 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 455890 T1122 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 455908 T1106 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 455914 T1122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 455918 T1122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 455919 T1122 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 455943 T1178 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 455950 T1122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 455951 T1122 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 455951 T1122 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 455951 T1122 oasc.SolrCore.<init> [testOverseerCol_shard1_replica1] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data/
   [junit4]   2> 455952 T1122 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fe1526c
   [junit4]   2> 455953 T1122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data
   [junit4]   2> 455954 T1122 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data/index/
   [junit4]   2> 455954 T1122 oasc.SolrCore.initIndex WARN [testOverseerCol_shard1_replica1] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 455955 T1122 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391399177179/testOverseerCol_shard1_replica1/data/index
   [junit4]   2> 455955 T1122 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=82.4609375, floorSegmentMB=1.365234375, forceMergeDeletesPctAllowed=7.33938087738324, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.20062408356616307
   [junit4]   2> 455956 T1122 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@586fb801 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@31dd55ff),segFN=segments_1,generation=1}
   [junit4]   2> 455957 T1122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 455970 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 455971 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 455971 T1122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 455972 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 455972 T1122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 455972 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 455973 T1122 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 455973 T1122 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 455974 T1122 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 455974 T1122 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 455988 T1122 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 455989 T1122 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 455990 T1122 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 455991 T1122 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 455992 T1122 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 455992 T1122 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 455993 T1122 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 456014 T1106 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 456021 T1106 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 456029 T1178 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 456033 T1178 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 456056 T1122 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 456062 T1122 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 456063 T1122 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 456064 T1122 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=88.7353515625, floorSegmentMB=1.6591796875, forceMergeDeletesPctAllowed=16.59613178711819, segmentsPerTier=30.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.28160697878444846
   [junit4]   2> 456089 T1106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 456096 T1178 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 456097 T1122 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@586fb801 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@31dd55ff),segFN=segments_1,generation=1}
   [junit4]   2> 456097 T1122 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 456098 T1122 oass.SolrIndexSearcher.<init> Opening Searcher@377b6d55 main
   [junit4]   2> 456106 T1196 oasc.SolrCore.registerSearcher [testOverseerCol_shard1_replica1] Registered new searcher Searcher@377b6d55 main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 456112 T1106 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 456116 T1178 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 456124 T1106 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 456127 T1117 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 456133 T1178 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 456140 T1106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 456147 T1106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 456147 T1106 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 456150 T1117 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 456154 T1178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 456155 T1178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 456155 T1178 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 456157 T1122 oasc.CoreContainer.registerCore registering core: testOverseerCol_shard1_replica1
   [junit4]   2> 456158 T1122 oasc.ZkController.register Register replica - core:testOverseerCol_shard1_replica1 address:http://127.0.0.1:49514 collection:testOverseerCol shard:shard1
   [junit4]   2> 456160 T1106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 456160 T1106 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 456160 T1106 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 456161 T1106 oasc.SolrCore.<init> [testOverseerCol_shard3_replica2] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2/data/
   [junit4]   2> 456161 T1106 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3fe1526c
   [junit4]   2> 456162 T1178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 456163 T1178 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 456163 T1106 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391399175490/testOverseerCol_shard3_replica2/data
   [junit4]   2> 456165 T1117 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 456163 T1178 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 456170 T1178 oasc.SolrCore.<init> [testOverseerCol_shard1_replica2] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391399184178/testOverseerCol_shard1_replica2/, dataDir=../../../../../../../../../Users/jenkins/works

[...truncated too long message...]

464653 T1168 C448 P49523 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
   [junit4]   2> 464653 T1168 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 464653 T1168 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49523/testOverseerCol_shard3_replica1/ shard3
   [junit4]   2> 464654 T1168 oasc.SolrCore.close [testOverseerCol_shard3_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@34ff584e
   [junit4]   2> 464669 T1168 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 464669 T1168 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 464669 T1168 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 464670 T1168 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 464671 T1168 oasc.SolrCore.closeSearcher [testOverseerCol_shard3_replica1] Closing main searcher on request.
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data [CachedDir<<refCount=0;path=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data;done=false>>]
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.closeCacheValue looking to close ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data/index [CachedDir<<refCount=0;path=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data/index;done=false>>]
   [junit4]   2> 464672 T1168 oasc.CachingDirectoryFactory.close Closing directory: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391399182428/testOverseerCol_shard3_replica1/data/index
   [junit4]   2> 464672 T1168 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leaders/shard3
   [junit4]   2> 464673 T1168 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 		at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 		at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:205)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:202)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:413)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:382)
   [junit4]   2> 		at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:369)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:112)
   [junit4]   2> 		at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:273)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 		at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 	
   [junit4]   2> 464673 T1168 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/testOverseerCol/leader_elect/shard3/election/91186736294920206-core_node3-n_0000000001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:173)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 464673 T1168 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene46: {}, docValues:{}, sim=DefaultSimilarity, locale=es_EC, timezone=Australia/Queensland
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=5,free=33196904,total=277872640
   [junit4]   2> NOTE: All tests run in this JVM: [BlockCacheTest, OverseerCollectionProcessorTest, MBeansHandlerTest, SignatureUpdateProcessorFactoryTest, TestDocSet, PreAnalyzedFieldTest, HdfsRecoveryZkTest, ResponseLogComponentTest, FullSolrCloudDistribCmdsTest, TermVectorComponentTest, TestExtendedDismaxParser, TestStressReorder, PolyFieldTest, InfoHandlerTest, SolrInfoMBeanTest, TestCopyFieldCollectionResource, RecoveryZkTest, TestLRUCache, TestFastWriter, MinimalSchemaTest, TestWriterPerf, TestDocBasedVersionConstraints, TimeZoneUtilsTest, SchemaVersionSpecificBehaviorTest, HdfsDirectoryTest, TestUniqueKeyFieldResource, DistanceFunctionTest, CurrencyFieldOpenExchangeTest, EnumFieldTest, TestElisionMultitermQuery, TestSolrQueryParserResource, DirectSolrSpellCheckerTest, DateMathParserTest, NoCacheHeaderTest, RequestHandlersTest, DistribCursorPagingTest, TestPerFieldSimilarity, TestNRTOpen, DefaultValueUpdateProcessorTest, LeaderElectionIntegrationTest, TestLMJelinekMercerSimilarityFactory, XmlUpdateRequestHandlerTest, CurrencyFieldXmlFileTest, ConnectionManagerTest, OverseerRolesTest]
   [junit4] Completed in 25.48s, 1 test, 1 error <<< FAILURES!

[...truncated 984 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: There were test failures: 365 suites, 1590 tests, 1 error, 40 ignored (6 assumptions)

Total time: 79 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.7.0) - Build # 1288 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/1288/
Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerRolesTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=10429, name=Overseer-91185229537607696-127.0.0.1:57506_ns_-n_0000000048, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=10429, name=Overseer-91185229537607696-127.0.0.1:57506_ns_-n_0000000048, state=RUNNABLE, group=Overseer collection creation process.]
Caused by: org.apache.solr.common.cloud.ZooKeeperException: 
	at __randomizedtesting.SeedInfo.seed([4E804FB2981567A]:0)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:202)
	at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /overseer/collection-queue-work/qnr-0000000004
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1266)
	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:290)
	at org.apache.solr.common.cloud.SolrZkClient$8.execute(SolrZkClient.java:287)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:287)
	at org.apache.solr.common.cloud.SolrZkClient.setData(SolrZkClient.java:473)
	at org.apache.solr.cloud.DistributedQueue.remove(DistributedQueue.java:190)
	at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:193)
	... 1 more




Build Log:
[...truncated 10746 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerRolesTest
   [junit4]   2> 2643211 T10205 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ns_/
   [junit4]   2> 2643215 T10205 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./solrtest-OverseerRolesTest-1391376183121
   [junit4]   2> 2643217 T10205 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 2643218 T10206 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 2643319 T10205 oasc.ZkTestServer.run start zk server on port:57484
   [junit4]   2> 2643325 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2643343 T10212 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@416025e5 name:ZooKeeperConnection Watcher:127.0.0.1:57484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2643345 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2643348 T10205 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 2643373 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2643383 T10214 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@547bd92a name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2643385 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2643385 T10205 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 2643404 T10205 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 2643416 T10205 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 2643423 T10205 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 2643434 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2643435 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2643449 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2643449 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 2643457 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2643458 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2643464 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2643465 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2643473 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2643473 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2643481 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2643481 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 2643490 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2643491 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2643503 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2643503 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2643513 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2643514 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2643521 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2643522 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2643529 T10205 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2643530 T10205 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2643862 T10205 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2643870 T10205 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57487
   [junit4]   2> 2643871 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2643872 T10205 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2643872 T10205 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447
   [junit4]   2> 2643873 T10205 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/'
   [junit4]   2> 2643909 T10205 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/solr.xml
   [junit4]   2> 2643994 T10205 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2643995 T10205 oasc.CoreContainer.<init> New CoreContainer 2010746253
   [junit4]   2> 2643995 T10205 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/]
   [junit4]   2> 2643997 T10205 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2643997 T10205 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2643997 T10205 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2643998 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2643998 T10205 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2643998 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2643998 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2643999 T10205 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2643999 T10205 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2644008 T10205 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2644009 T10205 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2644009 T10205 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2644010 T10205 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57484/solr
   [junit4]   2> 2644010 T10205 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2644012 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2644016 T10225 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bdc6173 name:ZooKeeperConnection Watcher:127.0.0.1:57484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2644017 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2644022 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2644027 T10227 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b2dcd45 name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2644027 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2644032 T10205 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 2644041 T10205 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 2644050 T10205 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 2644055 T10205 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57487_ns_
   [junit4]   2> 2644057 T10205 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57487_ns_
   [junit4]   2> 2644064 T10205 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 2644070 T10205 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 2644081 T10205 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:57487_ns_
   [junit4]   2> 2644081 T10205 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 2644087 T10205 oasc.Overseer.start Overseer (id=91185229537607683-127.0.0.1:57487_ns_-n_0000000000) starting
   [junit4]   2> 2644096 T10205 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 2644106 T10229 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 2644107 T10205 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 2644108 T10229 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
   [junit4]   2> 2644112 T10205 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 2644117 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2644125 T10228 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 2644132 T10205 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447
   [junit4]   2> 2644239 T10205 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/conf/
   [junit4]   2> 2644245 T10205 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2644246 T10205 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2644246 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2644248 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2644252 T10232 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@405ae950 name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2644253 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2644256 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2644262 T10205 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2644594 T10205 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 2644595 T10205 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2644598 T10205 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57491
   [junit4]   2> 2644602 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2644602 T10205 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2644603 T10205 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168
   [junit4]   2> 2644603 T10205 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168/'
   [junit4]   2> 2644631 T10205 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168/solr.xml
   [junit4]   2> 2644711 T10205 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2644712 T10205 oasc.CoreContainer.<init> New CoreContainer 1619103745
   [junit4]   2> 2644712 T10205 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168/]
   [junit4]   2> 2644714 T10205 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2644714 T10205 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2644715 T10205 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2644715 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2644716 T10205 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2644716 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2644716 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2644717 T10205 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2644717 T10205 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2644726 T10205 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2644727 T10205 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2644727 T10205 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2644728 T10205 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57484/solr
   [junit4]   2> 2644728 T10205 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2644730 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2644734 T10243 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a156326 name:ZooKeeperConnection Watcher:127.0.0.1:57484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2644734 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2644739 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2644743 T10245 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53da459d name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2644744 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2644761 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2645772 T10205 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57491_ns_
   [junit4]   2> 2645775 T10205 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57491_ns_
   [junit4]   2> 2645781 T10245 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2645782 T10227 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2645781 T10232 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 2645793 T10205 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168
   [junit4]   2> 2645893 T10205 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168/conf/
   [junit4]   2> 2645898 T10205 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2645899 T10205 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2645900 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2646227 T10205 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 2646227 T10205 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2646232 T10205 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57494
   [junit4]   2> 2646234 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2646234 T10205 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2646235 T10205 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391376185805
   [junit4]   2> 2646235 T10205 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391376185805/'
   [junit4]   2> 2646267 T10205 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391376185805/solr.xml
   [junit4]   2> 2646336 T10205 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2646336 T10205 oasc.CoreContainer.<init> New CoreContainer 604029235
   [junit4]   2> 2646336 T10205 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391376185805/]
   [junit4]   2> 2646338 T10205 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2646338 T10205 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2646338 T10205 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2646339 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2646339 T10205 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2646339 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2646339 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2646340 T10205 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2646340 T10205 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2646349 T10205 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2646350 T10205 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2646350 T10205 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2646351 T10205 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57484/solr
   [junit4]   2> 2646351 T10205 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2646353 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2646357 T10257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d8c0dbc name:ZooKeeperConnection Watcher:127.0.0.1:57484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2646358 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2646363 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2646366 T10259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d88d5e1 name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2646367 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2646377 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2647386 T10205 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57494_ns_
   [junit4]   2> 2647389 T10205 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57494_ns_
   [junit4]   2> 2647397 T10227 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2647398 T10259 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2647398 T10245 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2647397 T10232 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 2647410 T10205 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391376185805
   [junit4]   2> 2647565 T10205 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391376185805/conf/
   [junit4]   2> 2647574 T10205 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2647575 T10205 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2647576 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2647904 T10205 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 2647905 T10205 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2647909 T10205 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57497
   [junit4]   2> 2647911 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2647912 T10205 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2647912 T10205 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391376187481
   [junit4]   2> 2647913 T10205 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391376187481/'
   [junit4]   2> 2647944 T10205 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391376187481/solr.xml
   [junit4]   2> 2648021 T10205 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2648023 T10205 oasc.CoreContainer.<init> New CoreContainer 1449722006
   [junit4]   2> 2648023 T10205 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391376187481/]
   [junit4]   2> 2648024 T10205 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2648025 T10205 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2648025 T10205 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2648025 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2648026 T10205 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2648026 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2648027 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2648027 T10205 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2648027 T10205 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2648037 T10205 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2648038 T10205 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2648038 T10205 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2648039 T10205 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57484/solr
   [junit4]   2> 2648039 T10205 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2648040 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2648045 T10271 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a5a42c name:ZooKeeperConnection Watcher:127.0.0.1:57484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2648049 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2648056 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2648061 T10273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11ab68e6 name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2648063 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2648074 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2649087 T10205 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57497_ns_
   [junit4]   2> 2649092 T10205 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57497_ns_
   [junit4]   2> 2649106 T10273 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649106 T10259 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649107 T10227 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649106 T10232 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649108 T10245 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 2649140 T10205 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391376187481
   [junit4]   2> 2649245 T10205 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391376187481/conf/
   [junit4]   2> 2649251 T10205 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2649252 T10205 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2649252 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2649634 T10205 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 2649635 T10205 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2649641 T10205 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57500
   [junit4]   2> 2649641 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2649642 T10205 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2649642 T10205 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391376189158
   [junit4]   2> 2649643 T10205 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391376189158/'
   [junit4]   2> 2649676 T10205 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391376189158/solr.xml
   [junit4]   2> 2649748 T10205 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2649749 T10205 oasc.CoreContainer.<init> New CoreContainer 975732212
   [junit4]   2> 2649749 T10205 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391376189158/]
   [junit4]   2> 2649750 T10205 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2649751 T10205 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2649751 T10205 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2649751 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2649752 T10205 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2649752 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2649752 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2649753 T10205 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2649753 T10205 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2649762 T10205 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2649763 T10205 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2649763 T10205 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2649764 T10205 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57484/solr
   [junit4]   2> 2649765 T10205 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2649767 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2649776 T10285 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38653c00 name:ZooKeeperConnection Watcher:127.0.0.1:57484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2649777 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2649785 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2649790 T10287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55d675d4 name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2649790 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2649802 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2650816 T10205 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57500_ns_
   [junit4]   2> 2650818 T10205 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57500_ns_
   [junit4]   2> 2650826 T10273 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2650826 T10259 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2650826 T10227 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2650826 T10287 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2650826 T10245 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2650828 T10232 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 2650838 T10205 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391376189158
   [junit4]   2> 2650985 T10205 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391376189158/conf/
   [junit4]   2> 2650993 T10205 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2650994 T10205 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2650994 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2651372 T10205 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 5
   [junit4]   2> 2651373 T10205 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2651377 T10205 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57503
   [junit4]   2> 2651379 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2651379 T10205 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2651379 T10205 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391376190900
   [junit4]   2> 2651380 T10205 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391376190900/'
   [junit4]   2> 2651408 T10205 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391376190900/solr.xml
   [junit4]   2> 2651488 T10205 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2651489 T10205 oasc.CoreContainer.<init> New CoreContainer 650633034
   [junit4]   2> 2651489 T10205 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391376190900/]
   [junit4]   2> 2651490 T10205 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2651491 T10205 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2651491 T10205 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2651491 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2651492 T10205 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2651492 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2651493 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2651493 T10205 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2651494 T10205 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2651509 T10205 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2651509 T10205 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2651510 T10205 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2651510 T10205 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57484/solr
   [junit4]   2> 2651510 T10205 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2651513 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2651519 T10299 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c14872a name:ZooKeeperConnection Watcher:127.0.0.1:57484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2651520 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2651525 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2651528 T10301 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41e3fe27 name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2651529 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2651541 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2652551 T10205 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57503_ns_
   [junit4]   2> 2652554 T10205 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57503_ns_
   [junit4]   2> 2652563 T10245 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652563 T10301 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652563 T10232 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652564 T10227 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652564 T10287 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652563 T10273 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652563 T10259 oascc.ZkStateReader$3.process Updating live nodes... (6)
   [junit4]   2> 2652577 T10205 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391376190900
   [junit4]   2> 2652672 T10205 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391376190900/conf/
   [junit4]   2> 2652677 T10205 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2652678 T10205 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2652678 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2652984 T10205 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 6
   [junit4]   2> 2652985 T10205 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 2652988 T10205 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57506
   [junit4]   2> 2652990 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 2652991 T10205 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2652991 T10205 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391376192584
   [junit4]   2> 2652991 T10205 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391376192584/'
   [junit4]   2> 2653021 T10205 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391376192584/solr.xml
   [junit4]   2> 2653107 T10205 oasc.ConfigSolrXml.<init> Config-defined core root directory: 
   [junit4]   2> 2653108 T10205 oasc.CoreContainer.<init> New CoreContainer 396120358
   [junit4]   2> 2653108 T10205 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391376192584/]
   [junit4]   2> 2653109 T10205 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 2653110 T10205 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 2653110 T10205 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 2653110 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 2653110 T10205 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 2653111 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 2653111 T10205 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 2653111 T10205 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 2653112 T10205 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 2653120 T10205 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2653121 T10205 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2653121 T10205 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 2653122 T10205 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:57484/solr
   [junit4]   2> 2653122 T10205 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 2653124 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2653128 T10313 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1793a1e6 name:ZooKeeperConnection Watcher:127.0.0.1:57484 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2653129 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2653134 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2653138 T10315 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3beb18f1 name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2653138 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2653149 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2654159 T10205 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57506_ns_
   [junit4]   2> 2654161 T10205 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57506_ns_
   [junit4]   2> 2654168 T10227 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654168 T10245 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654169 T10315 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654169 T10232 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654169 T10259 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654169 T10273 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654169 T10301 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654169 T10287 oascc.ZkStateReader$3.process Updating live nodes... (7)
   [junit4]   2> 2654183 T10205 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391376192584
   [junit4]   2> 2654326 T10205 oasc.CorePropertiesLocator.discoverUnder Found core conf in /Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391376192584/conf/
   [junit4]   2> 2654333 T10205 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 2654334 T10205 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0
   [junit4]   2> 2654334 T10205 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 2654336 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2654340 T10318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57a9d7f2 name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2654341 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2654344 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2654353 T10205 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 2654358 T10320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@677a660d name:ZooKeeperConnection Watcher:127.0.0.1:57484/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2654359 T10205 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 2654362 T10205 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 2654373 T10236 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&version=2&numShards=4&name=testOverseerCol&maxShardsPerNode=2&action=CREATE&wt=javabin
   [junit4]   2> 2654375 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654377 T10229 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 2654378 T10229 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2> 	  "operation":"createcollection",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"testOverseerCol",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "maxShardsPerNode":"2"}
   [junit4]   2> 2654379 T10229 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 2654379 T10229 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/testOverseerCol 
   [junit4]   2> 2654380 T10229 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol
   [junit4]   2> 2654390 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654395 T10228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2654396 T10228 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: testOverseerCol
   [junit4]   2> 2654396 T10228 oasc.Overseer$ClusterStateUpdater.createCollection Create collection testOverseerCol with shards [shard1, shard2, shard3, shard4]
   [junit4]   2> 2654403 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654405 T10273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654405 T10227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654405 T10318 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654406 T10320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654406 T10259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654405 T10232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654406 T10245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654405 T10287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654405 T10315 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654405 T10301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654491 T10229 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2, shard3, shard4] , repFactor : 2
   [junit4]   2> 2654492 T10229 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica1 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:57491_ns_
   [junit4]   2> 2654492 T10229 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard1_replica2 as part of slice shard1 of collection testOverseerCol on 127.0.0.1:57497_ns_
   [junit4]   2> 2654493 T10229 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica1 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:57494_ns_
   [junit4]   2> 2654496 T10229 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard2_replica2 as part of slice shard2 of collection testOverseerCol on 127.0.0.1:57506_ns_
   [junit4]   2> 2654497 T10229 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica1 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:57500_ns_
   [junit4]   2> 2654499 T10229 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard3_replica2 as part of slice shard3 of collection testOverseerCol on 127.0.0.1:57503_ns_
   [junit4]   2> 2654501 T10237 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica1 state=down
   [junit4]   2> 2654502 T10229 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica1 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:57487_ns_
   [junit4]   2> 2654504 T10253 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica1 state=down
   [junit4]   2> 2654505 T10310 oasc.ZkController.publish publishing core=testOverseerCol_shard2_replica2 state=down
   [junit4]   2> 2654506 T10229 oasc.OverseerCollectionProcessor.createCollection Creating shard testOverseerCol_shard4_replica2 as part of slice shard4 of collection testOverseerCol on 127.0.0.1:57491_ns_
   [junit4]   2> 2654511 T10253 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654513 T10220 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica1 state=down
   [junit4]   2> 2654516 T10310 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654509 T10237 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654509 T10293 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica2 state=down
   [junit4]   2> 2654518 T10238 oasc.ZkController.publish publishing core=testOverseerCol_shard4_replica2 state=down
   [junit4]   2> 2654508 T10268 oasc.ZkController.publish publishing core=testOverseerCol_shard1_replica2 state=down
   [junit4]   2> 2654512 T10278 oasc.ZkController.publish publishing core=testOverseerCol_shard3_replica1 state=down
   [junit4]   2> 2654521 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654524 T10238 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654524 T10278 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654524 T10293 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654524 T10220 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654524 T10268 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 2654530 T10228 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
   [junit4]   2> 2654531 T10228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57491/ns_",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57491_ns_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654536 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654537 T10228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57494/ns_",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57494_ns_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654543 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654552 T10228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57506/ns_",
   [junit4]   2> 	  "core":"testOverseerCol_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57506_ns_",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654560 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654562 T10228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57500/ns_",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57500_ns_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654569 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654571 T10228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57491/ns_",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57491_ns_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654577 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654580 T10228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57503/ns_",
   [junit4]   2> 	  "core":"testOverseerCol_shard3_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57503_ns_",
   [junit4]   2> 	  "shard":"shard3",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654585 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654587 T10228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57497/ns_",
   [junit4]   2> 	  "core":"testOverseerCol_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57497_ns_",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654593 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654596 T10228 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=4 message={
   [junit4]   2> 	  "operation":"state",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:57487/ns_",
   [junit4]   2> 	  "core":"testOverseerCol_shard4_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "node_name":"127.0.0.1:57487_ns_",
   [junit4]   2> 	  "shard":"shard4",
   [junit4]   2> 	  "shard_range":null,
   [junit4]   2> 	  "shard_state":"active",
   [junit4]   2> 	  "shard_parent":null,
   [junit4]   2> 	  "collection":"testOverseerCol",
   [junit4]   2> 	  "numShards":"4",
   [junit4]   2> 	  "core_node_name":null}
   [junit4]   2> 2654602 T10227 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 2654709 T10320 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654709 T10259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654710 T10318 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654710 T10301 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654710 T10273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654710 T10287 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654709 T10227 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654710 T10245 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654709 T10315 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2654709 T10232 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
   [junit4]   2> 2655514 T10253 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica1
   [junit4]   2> 2655517 T10253 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391376185805/testOverseerCol_shard2_replica1
   [junit4]   2> 2655517 T10253 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655518 T10310 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard2_replica2
   [junit4]   2> 2655518 T10237 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica1
   [junit4]   2> 2655520 T10253 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655520 T10253 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655521 T10237 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168/testOverseerCol_shard1_replica1
   [junit4]   2> 2655522 T10237 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655520 T10310 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard2_replica2' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391376192584/testOverseerCol_shard2_replica2
   [junit4]   2> 2655523 T10310 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655523 T10237 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655523 T10237 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655522 T10253 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty2-1391376185805/testOverseerCol_shard2_replica1/'
   [junit4]   2> 2655524 T10310 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655524 T10310 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655526 T10238 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica2
   [junit4]   2> 2655530 T10238 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168/testOverseerCol_shard4_replica2
   [junit4]   2> 2655531 T10238 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655532 T10278 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica1
   [junit4]   2> 2655534 T10278 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391376189158/testOverseerCol_shard3_replica1
   [junit4]   2> 2655535 T10278 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655535 T10293 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard3_replica2
   [junit4]   2> 2655537 T10293 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard3_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty5-1391376190900/testOverseerCol_shard3_replica2
   [junit4]   2> 2655538 T10293 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655538 T10220 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard4_replica1
   [junit4]   2> 2655540 T10220 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard4_replica1' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/testOverseerCol_shard4_replica1
   [junit4]   2> 2655540 T10220 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655541 T10268 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for testOverseerCol_shard1_replica2
   [junit4]   2> 2655542 T10268 oasc.CoreContainer.create Creating SolrCore 'testOverseerCol_shard1_replica2' using instanceDir: ./org.apache.solr.cloud.OverseerRolesTest-jetty3-1391376187481/testOverseerCol_shard1_replica2
   [junit4]   2> 2655543 T10268 oasc.ZkController.createCollectionZkNode Check for collection zkNode:testOverseerCol
   [junit4]   2> 2655546 T10238 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655547 T10238 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655547 T10220 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655547 T10220 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655548 T10278 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655548 T10278 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655548 T10293 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655548 T10293 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655549 T10268 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 2655549 T10268 oascc.ZkStateReader.readConfigName Load collection config from:/collections/testOverseerCol
   [junit4]   2> 2655552 T10310 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty6-1391376192584/testOverseerCol_shard2_replica2/'
   [junit4]   2> 2655558 T10237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168/testOverseerCol_shard1_replica1/'
   [junit4]   2> 2655577 T10268 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty3-1391376187481/testOverseerCol_shard1_replica2/'
   [junit4]   2> 2655582 T10293 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty5-1391376190900/testOverseerCol_shard3_replica2/'
   [junit4]   2> 2655577 T10278 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.OverseerRolesTest-jetty4-1391376189158/testOverseerCol_shard3_replica1/'
   [junit4]   2> 2655589 T10220 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/testOverseerCol_shard4_replica1/'
   [junit4]   2> 2655596 T10238 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.OverseerRolesTest-jetty1-1391376184168/testOverseerCol_shard4_replica2/'
   [junit4]   2> 2655763 T10278 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2655764 T10310 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2655778 T10237 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2655804 T10253 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2655805 T10268 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2655856 T10293 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2655861 T10238 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2655959 T10220 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
   [junit4]   2> 2655995 T10268 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656009 T10268 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656018 T10268 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica2] Schema name=test
   [junit4]   2> 2656054 T10253 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656071 T10278 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656080 T10278 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656103 T10253 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656114 T10253 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica1] Schema name=test
   [junit4]   2> 2656128 T10238 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656128 T10310 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656131 T10293 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656139 T10310 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656142 T10237 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656146 T10278 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica1] Schema name=test
   [junit4]   2> 2656150 T10310 oass.IndexSchema.readSchema [testOverseerCol_shard2_replica2] Schema name=test
   [junit4]   2> 2656152 T10293 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656156 T10238 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656158 T10237 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656168 T10293 oass.IndexSchema.readSchema [testOverseerCol_shard3_replica2] Schema name=test
   [junit4]   2> 2656176 T10237 oass.IndexSchema.readSchema [testOverseerCol_shard1_replica1] Schema name=test
   [junit4]   2> 2656218 T10238 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica2] Schema name=test
   [junit4]   2> 2656280 T10220 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2656283 T10220 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 2656292 T10220 oass.IndexSchema.readSchema [testOverseerCol_shard4_replica1] Schema name=test
   [junit4]   2> 2658455 T10220 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2658525 T10220 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2658530 T10220 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2658596 T10310 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2658611 T10220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2658651 T10293 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2658658 T10293 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2658674 T10268 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2658680 T10293 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2658695 T10220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2658698 T10310 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2658705 T10310 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2658712 T10220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2658715 T10268 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2658718 T10220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2658718 T10220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2658719 T10220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2658722 T10220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2658722 T10220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2658723 T10220 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2658724 T10220 oasc.SolrCore.<init> [testOverseerCol_shard4_replica1] Opening new SolrCore at ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/testOverseerCol_shard4_replica1/, dataDir=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/testOverseerCol_shard4_replica1/data/
   [junit4]   2> 2658725 T10220 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b2457bb
   [junit4]   2> 2658727 T10220 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/testOverseerCol_shard4_replica1/data
   [junit4]   2> 2658727 T10220 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/testOverseerCol_shard4_replica1/data/index/
   [junit4]   2> 2658727 T10220 oasc.SolrCore.initIndex WARN [testOverseerCol_shard4_replica1] Solr index directory './org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/testOverseerCol_shard4_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2658728 T10220 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.OverseerRolesTest-controljetty-1391376183447/testOverseerCol_shard4_replica1/data/index
   [junit4]   2> 2658729 T10220 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1591612991, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 2658730 T10220 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@284f689f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@db8e935),segFN=segments_1,generation=1}
   [junit4]   2> 2658730 T10220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2658737 T10220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2658743 T10220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2658743 T10220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2658744 T10220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2658744 T10220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2658745 T10220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2658745 T10220 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2658746 T10220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2658746 T10220 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2658747 T10220 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 2658747 T10220 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 2658747 T10220 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 2658748 T10220 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 2658748 T10220 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 2658749 T10220 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 2658751 T10220 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 2658751 T10220 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
   [junit4]   2> 2658773 T10220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 2658779 T10253 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2658783 T10293 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2658788 T10220 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 2658790 T10220 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 2658790 T10253 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 2658794 T10268 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2658803 T10293 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 2658804 T10220 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1206696151606538]
   [junit4]   2> 2658814 T10293 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2658794 T10253 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 2658824 T10220 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(org.apache.lucene.store.RAMDirectory@284f689f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@db8e935),segFN=segments_1,generation=1}
   [junit4]   2> 2658825 T10220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 2658825 T10220 oass.SolrIndexSearcher.<init> Opening Searcher@44c4ee0a main
   [junit4]   2> 2658833 T10293 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2658834 T10293 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2658835 T10329 oasc.SolrCore.registerSearcher [testOverseerCol_shard4_replica1] Registered new searcher Searcher@44c4ee0a main{StandardDirectoryReader(segments_1:1:nrt)}
   [junit4]   2> 2658837 T10293 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2658837 T10220 oasc.CoreContainer.registerCore registering core: testOverseerCol_shard4_replica1
   [junit4]   2> 2658840 T10220 oasc.ZkController.register Register replica - core:testOverseerCol_shard4_replica1 address:http://127.0.0.1:57487/ns_ collection:testOverseerCol shard:shard4
   [junit4]   2> 2658838 T10237 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2658846 T10220 oascc.SolrZkClient.makePath makePath: /collections/testOverseerCol/leader_elect/shard4/election
   [junit4]   2> 2658843 T10293 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 2658855 T10293 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 2658856 T10293 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2658856 T10293 oasc.SolrCore.<init> [testOverseerCol_shard3_replica2] Opening new SolrCor

[...truncated too long message...]

2> 	at org.apache.solr.common.cloud.SolrZkClient$2.execute(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:170)
   [junit4]   2> 	at org.apache.solr.cloud.ElectionContext.cancelElection(ElectionContext.java:67)
   [junit4]   2> 	at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:293)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:164)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:108)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2> 
   [junit4]   2> 2668260 T10273 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=fr_CH, timezone=Australia/Brisbane
   [junit4]   2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_51 (64-bit)/cpus=2,threads=2,free=139848368,total=443023360
   [junit4]   2> NOTE: All tests run in this JVM: [TestFieldResource, DisMaxRequestHandlerTest, PreAnalyzedFieldTest, DirectSolrConnectionTest, StatelessScriptUpdateProcessorFactoryTest, DistributedSuggestComponentTest, FullSolrCloudDistribCmdsTest, TestZkChroot, TestReloadAndDeleteDocs, DistributedDebugComponentTest, FastVectorHighlighterTest, TermsComponentTest, TestLRUCache, HdfsCollectionsAPIDistributedZkTest, TestRangeQuery, TestNonNRTOpen, QueryFacetTest, TestRealTimeGet, SolrXmlInZkTest, TestSchemaVersionResource, MinimalSchemaTest, TestCSVLoader, SortByFunctionTest, TestCharFilters, NoCacheHeaderTest, RangeFacetTest, HdfsBasicDistributedZkTest, TestFreeTextSuggestions, TestSolrDeletionPolicy2, DefaultValueUpdateProcessorTest, SignatureUpdateProcessorFactoryTest, QueryElevationComponentTest, TestDefaultSimilarityFactory, TestDistribDocBasedVersion, TestStressLucene, PreAnalyzedUpdateProcessorTest, ShardRoutingTest, ParsingFieldUpdateProcessorsTest, TestSolrJ, TestBinaryResponseWriter, BadCopyFieldTest, DistanceFunctionTest, TestDynamicFieldResource, TestDocBasedVersionConstraints, DocValuesMultiTest, BlockCacheTest, CircularListTest, TestJmxIntegration, TestSchemaSimilarityResource, TestFiltering, DOMUtilTest, TestSolrDeletionPolicy1, DocumentBuilderTest, TimeZoneUtilsTest, TestPhraseSuggestions, ClusterStateUpdateTest, TestDistributedMissingSort, DebugComponentTest, XsltUpdateRequestHandlerTest, SuggesterTSTTest, DateFieldTest, HighlighterTest, RegexBoostProcessorTest, SimpleFacetsTest, DocValuesMissingTest, SoftAutoCommitTest, LukeRequestHandlerTest, TestSolrXmlPersistor, SuggesterTest, PluginInfoTest, StandardRequestHandlerTest, BasicDistributedZk2Test, DocValuesTest, TestAtomicUpdateErrorCases, TestDynamicFieldCollectionResource, AbstractAnalyticsFacetTest, TestNumberUtils, JsonLoaderTest, TestUniqueKeyFieldResource, BlockDirectoryTest, TestLFUCache, SolrInfoMBeanTest, SchemaVersionSpecificBehaviorTest, ZkCLITest, SolrCoreTest, MoreLikeThisHandlerTest, SpatialFilterTest, CoreAdminCreateDiscoverTest, TestClassNameShortening, AliasIntegrationTest, TestXIncludeConfig, DirectUpdateHandlerTest, TestIndexingPerformance, TestLuceneMatchVersion, TestSchemaNameResource, CurrencyFieldOpenExchangeTest, TestCollapseQParserPlugin, RecoveryZkTest, OutputWriterTest, BasicDistributedZkTest, TestDocumentBuilder, TestSchemaResource, TestFieldCollectionResource, UpdateRequestProcessorFactoryTest, StatsComponentTest, TestBadConfig, FileBasedSpellCheckerTest, SolrIndexSplitterTest, TestCollationField, NoFacetTest, AbstractAnalyticsStatsTest, AddSchemaFieldsUpdateProcessorFactoryTest, RequestHandlersTest, TestMergePolicyConfig, LegacyHTMLStripCharFilterTest, IndexBasedSpellCheckerTest, HdfsBasicDistributedZk2Test, HdfsChaosMonkeySafeLeaderTest, OverseerTest, ZkNodePropsTest, TestGroupingSearch, UUIDFieldTest, TestSolrQueryParserDefaultOperatorResource, SuggesterFSTTest, PathHierarchyTokenizerFactoryTest, TestCoreContainer, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, TestRTGBase, CustomCollectionTest, SolrRequestParserTest, SpellCheckCollatorTest, TestFoldingMultitermQuery, TestSweetSpotSimilarityFactory, SuggestComponentTest, PolyFieldTest, TestExtendedDismaxParser, TestPerFieldSimilarity, TestSearchPerf, TestSerializedLuceneMatchVersion, ExternalFileFieldSortTest, TestFaceting, TestShardHandlerFactory, PrimUtilsTest, SpellCheckComponentTest, TestFastLRUCache, HighlighterConfigTest, TestLazyCores, TestStressVersions, TestQuerySenderListener, TestDistributedSearch, CoreAdminHandlerTest, TestFastOutputStream, TestAddFieldRealTimeGet, ShowFileRequestHandlerTest, TestHashPartitioner, DistribCursorPagingTest, TestSolrXml, HdfsDirectoryTest, CurrencyFieldXmlFileTest, TestSimpleQParserPlugin, TestSolrQueryParserResource, TestConfig, TestPHPSerializedResponseWriter, TestFieldTypeCollectionResource, TestDistributedGrouping, TestAnalyzedSuggestions, FieldAnalysisRequestHandlerTest, RequiredFieldsTest, TestSolr4Spatial, TestUpdate, TestIndexSearcher, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, LeaderElectionIntegrationTest, BasicZkTest, TestReplicationHandler, TestRandomFaceting, LeaderElectionTest, ZkSolrClientTest, TestRandomDVFaceting, ShardRoutingCustomTest, SliceStateUpdateTest, AutoCommitTest, TestRecovery, ZkControllerTest, TestStressReorder, TestJoin, TestMultiCoreConfBootstrap, TestReload, HardAutoCommitTest, DistributedTermsComponentTest, SolrCmdDistributorTest, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestSort, TestFunctionQuery, BasicFunctionalityTest, OverseerCollectionProcessorTest, DistributedQueryElevationComponentTest, SimplePostToolTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, TestTrie, SuggesterWFSTTest, WordBreakSolrSpellCheckerTest, SolrCoreCheckLockOnStartupTest, TestPseudoReturnFields, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, TestRemoteStreaming, CacheHeaderTest, TestSurroundQueryParser, TestQueryUtils, TestWriterPerf, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, TestQueryTypes, PrimitiveFieldTypeTest, TestOmitPositions, XmlUpdateRequestHandlerTest, TestValueSourceCache, TermVectorComponentTest, TestSolrQueryParser, TestArbitraryIndexDir, IndexSchemaRuntimeFieldTest, LoggingHandlerTest, SolrPluginUtilsTest, IndexSchemaTest, ReturnFieldsTest, JSONWriterTest, TestCSVResponseWriter, MBeansHandlerTest, QueryParsingTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, CSVRequestHandlerTest, TestComponentsName, SearchHandlerTest, CoreContainerCoreInitFailuresTest, SOLR749Test, UpdateParamsTest, AlternateDirectoryTest, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, MultiTermTest, SampleTest, TestDocSet, TestBinaryField, NumericFieldsTest, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, TestPostingsSolrHighlighter, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, TestCodecSupport, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestLMJelinekMercerSimilarityFactory, TestIBSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TestJmxMonitoredMap, ResourceLoaderTest, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, TestSolrXMLSerializer, TestSystemIdResolver, TestSuggestSpellingConverter, SpellingQueryConverterTest, RAMDirectoryFactoryTest, ClusterStateTest, TestUtils, SliceStateTest, SystemInfoHandlerTest, FileUtilsTest, CursorPagingTest, SolrTestCaseJ4Test, TestCursorMarkWithoutUniqueKey, ExpressionTest, FieldFacetExtrasTest, FunctionTest, AssignTest, ConnectionManagerTest, DeleteReplicaTest, DeleteShardTest, MigrateRouteKeyTest, OverseerRolesTest]
   [junit4] Completed in 25.17s, 1 test, 1 error <<< FAILURES!

[...truncated 137 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:453: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:433: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/extra-targets.xml:37: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/solr/common-build.xml:491: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:1268: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-trunk-MacOSX/lucene/common-build.xml:901: There were test failures: 365 suites, 1590 tests, 1 error, 40 ignored (6 assumptions)

Total time: 81 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure