You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/07/01 07:48:04 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b94) - Build #
6315 - Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6315/
Java: 64bit/jdk1.8.0-ea-b94 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([67E9A14DA93DFB66]:0)
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([67E9A14DA93DFB66]:0)
Build Log:
[...truncated 10293 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 489864 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!! instance=1040410215
[junit4:junit4] 2> 489868 T1701 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /w/ai
[junit4:junit4] 2> 489872 T1701 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1372650448063
[junit4:junit4] 2> 489872 T1701 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 489873 T1702 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 489973 T1701 oasc.ZkTestServer.run start zk server on port:40839
[junit4:junit4] 2> 489974 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 490095 T1708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71521c3f name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 490095 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 490095 T1701 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 490100 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 490101 T1710 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f95ec65 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 490101 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 490102 T1701 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 490104 T1701 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 490106 T1701 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 490107 T1701 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 490109 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 490109 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 490112 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 490113 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 490115 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 490115 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 490117 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 490118 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 490119 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 490120 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 490122 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 490122 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 490124 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 490124 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 490126 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 490126 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 490128 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 490129 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 490194 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 490211 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42089
[junit4:junit4] 2> 490211 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 490212 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 490212 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322
[junit4:junit4] 2> 490212 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/solr.xml
[junit4:junit4] 2> 490212 T1701 oasc.CoreContainer.<init> New CoreContainer 554851461
[junit4:junit4] 2> 490213 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/'
[junit4:junit4] 2> 490213 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/'
[junit4:junit4] 2> 490252 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 490253 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 490253 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 490253 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 490253 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 490254 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 490254 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 490254 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 490254 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 490255 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 490257 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 490258 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 490258 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 490259 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 490261 T1721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e210446 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 490261 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 490263 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 490265 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 490267 T1723 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b4e59d5 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 490267 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 490268 T1701 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 490271 T1701 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 490273 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 490274 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42089_w%2Fai
[junit4:junit4] 2> 490275 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42089_w%2Fai
[junit4:junit4] 2> 490277 T1701 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 490282 T1701 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 490284 T1701 oasc.Overseer.start Overseer (id=89958019764387843-127.0.0.1:42089_w%2Fai-n_0000000000) starting
[junit4:junit4] 2> 490286 T1701 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 490289 T1725 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 490289 T1701 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 490291 T1701 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 490292 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 490294 T1724 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 490296 T1726 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 490296 T1726 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 491797 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 491798 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42089_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42089/w/ai"}
[junit4:junit4] 2> 491798 T1724 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 491798 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 491804 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 492297 T1726 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1
[junit4:junit4] 2> 492298 T1726 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 492298 T1726 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 492299 T1726 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 492300 T1726 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1/'
[junit4:junit4] 2> 492300 T1726 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1/lib/README' to classloader
[junit4:junit4] 2> 492301 T1726 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 492332 T1726 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 492372 T1726 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 492374 T1726 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 492377 T1726 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 492707 T1726 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 492708 T1726 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 492709 T1726 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 492718 T1726 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 492721 T1726 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 492766 T1726 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 492771 T1726 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 492776 T1726 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 492778 T1726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 492778 T1726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 492779 T1726 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 492780 T1726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 492780 T1726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 492780 T1726 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 492781 T1726 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/
[junit4:junit4] 2> 492781 T1726 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
[junit4:junit4] 2> 492781 T1726 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 492782 T1726 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data
[junit4:junit4] 2> 492782 T1726 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/index/
[junit4:junit4] 2> 492783 T1726 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 492783 T1726 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/index
[junit4:junit4] 2> 492788 T1726 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 492788 T1726 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 492790 T1726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 492790 T1726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 492791 T1726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 492792 T1726 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 492792 T1726 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 492792 T1726 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 492793 T1726 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 492793 T1726 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 492793 T1726 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 492796 T1726 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 492800 T1726 oass.SolrIndexSearcher.<init> Opening Searcher@4b4def74 main
[junit4:junit4] 2> 492801 T1726 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 492801 T1726 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 492805 T1726 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 492806 T1726 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42089/w/ai collection:control_collection shard:shard1
[junit4:junit4] 2> 492806 T1726 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 492807 T1727 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b4def74 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 492816 T1726 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 492819 T1726 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 492819 T1726 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 492819 T1726 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42089/w/ai/collection1/
[junit4:junit4] 2> 492820 T1726 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 492820 T1726 oasc.SyncStrategy.syncToMe http://127.0.0.1:42089/w/ai/collection1/ has no replicas
[junit4:junit4] 2> 492820 T1726 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42089/w/ai/collection1/
[junit4:junit4] 2> 492820 T1726 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 493309 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 493319 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 493328 T1726 oasc.ZkController.register We are http://127.0.0.1:42089/w/ai/collection1/ and leader is http://127.0.0.1:42089/w/ai/collection1/
[junit4:junit4] 2> 493329 T1726 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42089/w/ai
[junit4:junit4] 2> 493329 T1726 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 493329 T1726 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 493329 T1726 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 493331 T1726 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 493333 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 493333 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 493334 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 493338 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 493339 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 493341 T1730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8c7606d name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 493342 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 493346 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 493352 T1701 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 493469 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 493484 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50526
[junit4:junit4] 2> 493488 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 493488 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 493488 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543
[junit4:junit4] 2> 493489 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/solr.xml
[junit4:junit4] 2> 493489 T1701 oasc.CoreContainer.<init> New CoreContainer 1314004103
[junit4:junit4] 2> 493490 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/'
[junit4:junit4] 2> 493490 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/'
[junit4:junit4] 2> 493541 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 493542 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 493542 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 493543 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 493543 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 493543 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 493544 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 493544 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 493544 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 493545 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 493548 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 493548 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 493549 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 493549 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 493551 T1743 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74be1c42 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 493551 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 493553 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 493556 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 493558 T1745 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@767d49cc name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 493558 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 493568 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 494573 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50526_w%2Fai
[junit4:junit4] 2> 494574 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50526_w%2Fai
[junit4:junit4] 2> 494578 T1730 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 494578 T1723 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 494579 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 494579 T1745 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 494585 T1746 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 494585 T1746 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 494825 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 494826 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42089_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42089/w/ai"}
[junit4:junit4] 2> 494830 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50526_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50526/w/ai"}
[junit4:junit4] 2> 494830 T1724 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 494831 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 494837 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 494837 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 494837 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 495588 T1746 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1
[junit4:junit4] 2> 495588 T1746 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 495589 T1746 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 495589 T1746 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 495590 T1746 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1/'
[junit4:junit4] 2> 495591 T1746 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1/lib/README' to classloader
[junit4:junit4] 2> 495591 T1746 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 495619 T1746 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 495645 T1746 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 495647 T1746 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 495649 T1746 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 495945 T1746 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 495945 T1746 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 495946 T1746 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 495950 T1746 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 495952 T1746 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 495966 T1746 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 495970 T1746 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 495973 T1746 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 495974 T1746 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 495975 T1746 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 495975 T1746 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 495976 T1746 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 495976 T1746 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 495976 T1746 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 495976 T1746 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/
[junit4:junit4] 2> 495977 T1746 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
[junit4:junit4] 2> 495977 T1746 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 495977 T1746 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1
[junit4:junit4] 2> 495977 T1746 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index/
[junit4:junit4] 2> 495978 T1746 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 495978 T1746 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index
[junit4:junit4] 2> 495981 T1746 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 495981 T1746 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 495982 T1746 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 495983 T1746 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 495983 T1746 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 495984 T1746 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 495984 T1746 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 495984 T1746 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 495984 T1746 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 495985 T1746 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 495985 T1746 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 495986 T1746 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 495989 T1746 oass.SolrIndexSearcher.<init> Opening Searcher@3d61bd6d main
[junit4:junit4] 2> 495989 T1746 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 495990 T1746 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 495993 T1747 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3d61bd6d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 495995 T1746 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 495995 T1746 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50526/w/ai collection:collection1 shard:shard1
[junit4:junit4] 2> 495996 T1746 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 496001 T1746 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 496003 T1746 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 496003 T1746 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 496003 T1746 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50526/w/ai/collection1/
[junit4:junit4] 2> 496003 T1746 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 496004 T1746 oasc.SyncStrategy.syncToMe http://127.0.0.1:50526/w/ai/collection1/ has no replicas
[junit4:junit4] 2> 496004 T1746 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50526/w/ai/collection1/
[junit4:junit4] 2> 496004 T1746 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 496344 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 496352 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 496352 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 496352 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 496359 T1746 oasc.ZkController.register We are http://127.0.0.1:50526/w/ai/collection1/ and leader is http://127.0.0.1:50526/w/ai/collection1/
[junit4:junit4] 2> 496359 T1746 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50526/w/ai
[junit4:junit4] 2> 496360 T1746 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 496360 T1746 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 496360 T1746 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 496361 T1746 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 496363 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 496363 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 496363 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 496429 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 496431 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42132
[junit4:junit4] 2> 496432 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 496432 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 496432 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557
[junit4:junit4] 2> 496433 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/solr.xml
[junit4:junit4] 2> 496433 T1701 oasc.CoreContainer.<init> New CoreContainer 732481326
[junit4:junit4] 2> 496433 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/'
[junit4:junit4] 2> 496433 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/'
[junit4:junit4] 2> 496473 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 496475 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 496475 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 496475 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 496475 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 496478 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 496479 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 496479 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 496479 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 496481 T1759 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c76b072 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 496481 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 496483 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 496485 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 496487 T1761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ccc4d8e name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 496487 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 496490 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 497494 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42132_w%2Fai
[junit4:junit4] 2> 497495 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42132_w%2Fai
[junit4:junit4] 2> 497497 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 497498 T1723 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 497498 T1730 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 497498 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 497498 T1761 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 497498 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 497499 T1745 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 497503 T1762 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 497503 T1762 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 497857 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 497858 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50526_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50526/w/ai"}
[junit4:junit4] 2> 497862 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42132_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42132/w/ai"}
[junit4:junit4] 2> 497862 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 497862 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 497865 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 497865 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 497865 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 497865 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 498505 T1762 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1
[junit4:junit4] 2> 498505 T1762 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 498506 T1762 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 498506 T1762 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 498507 T1762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1/'
[junit4:junit4] 2> 498508 T1762 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1/lib/README' to classloader
[junit4:junit4] 2> 498508 T1762 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 498530 T1762 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 498559 T1762 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 498561 T1762 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 498563 T1762 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 498844 T1762 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 498844 T1762 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 498845 T1762 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 498850 T1762 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 498852 T1762 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 498863 T1762 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 498866 T1762 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 498868 T1762 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 498869 T1762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 498869 T1762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 498869 T1762 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 498870 T1762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 498870 T1762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 498871 T1762 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 498871 T1762 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/
[junit4:junit4] 2> 498871 T1762 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
[junit4:junit4] 2> 498871 T1762 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 498872 T1762 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2
[junit4:junit4] 2> 498872 T1762 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index/
[junit4:junit4] 2> 498872 T1762 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 498872 T1762 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index
[junit4:junit4] 2> 498875 T1762 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 498876 T1762 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 498877 T1762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 498878 T1762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 498878 T1762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 498878 T1762 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 498879 T1762 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 498879 T1762 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 498879 T1762 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 498879 T1762 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 498880 T1762 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 498881 T1762 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 498884 T1762 oass.SolrIndexSearcher.<init> Opening Searcher@1ffb2d75 main
[junit4:junit4] 2> 498884 T1762 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 498884 T1762 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 498887 T1763 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ffb2d75 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 498889 T1762 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 498889 T1762 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42132/w/ai collection:collection1 shard:shard2
[junit4:junit4] 2> 498890 T1762 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 498895 T1762 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 498897 T1762 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 498897 T1762 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 498897 T1762 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42132/w/ai/collection1/
[junit4:junit4] 2> 498897 T1762 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 498898 T1762 oasc.SyncStrategy.syncToMe http://127.0.0.1:42132/w/ai/collection1/ has no replicas
[junit4:junit4] 2> 498898 T1762 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42132/w/ai/collection1/
[junit4:junit4] 2> 498898 T1762 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 499371 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 499377 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 499377 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 499377 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 499377 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 499403 T1762 oasc.ZkController.register We are http://127.0.0.1:42132/w/ai/collection1/ and leader is http://127.0.0.1:42132/w/ai/collection1/
[junit4:junit4] 2> 499403 T1762 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42132/w/ai
[junit4:junit4] 2> 499403 T1762 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 499403 T1762 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 499404 T1762 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 499412 T1762 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 499413 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 499413 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 499413 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 499479 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 499487 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53140
[junit4:junit4] 2> 499488 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 499488 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 499489 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607
[junit4:junit4] 2> 499489 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/solr.xml
[junit4:junit4] 2> 499489 T1701 oasc.CoreContainer.<init> New CoreContainer 620715449
[junit4:junit4] 2> 499490 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/'
[junit4:junit4] 2> 499490 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/'
[junit4:junit4] 2> 499530 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 499530 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 499532 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 499532 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 499532 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 499535 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 499536 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 499536 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 499536 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 499538 T1775 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ab80171 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 499538 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 499542 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 499546 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 499547 T1777 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d809383 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 499547 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 499551 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 500553 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53140_w%2Fai
[junit4:junit4] 2> 500554 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53140_w%2Fai
[junit4:junit4] 2> 500556 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 500556 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 500557 T1723 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 500557 T1777 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 500557 T1730 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 500557 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 500558 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 500559 T1745 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 500559 T1761 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 500563 T1778 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 500563 T1778 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 500882 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 500883 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:42132_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:42132/w/ai"}
[junit4:junit4] 2> 500886 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53140_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53140/w/ai"}
[junit4:junit4] 2> 500886 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 500886 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 500889 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 500889 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 500889 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 500889 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 500889 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 501564 T1778 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1
[junit4:junit4] 2> 501564 T1778 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 501565 T1778 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 501565 T1778 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 501567 T1778 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1/'
[junit4:junit4] 2> 501567 T1778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1/lib/README' to classloader
[junit4:junit4] 2> 501567 T1778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 501594 T1778 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 501620 T1778 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 501621 T1778 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 501624 T1778 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 501923 T1778 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 501924 T1778 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 501925 T1778 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 501929 T1778 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 501931 T1778 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 501944 T1778 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 501948 T1778 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 501951 T1778 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 501952 T1778 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 501953 T1778 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 501953 T1778 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 501954 T1778 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 501954 T1778 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 501954 T1778 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 501954 T1778 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/
[junit4:junit4] 2> 501955 T1778 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
[junit4:junit4] 2> 501955 T1778 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 501955 T1778 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3
[junit4:junit4] 2> 501956 T1778 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index/
[junit4:junit4] 2> 501956 T1778 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 501956 T1778 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index
[junit4:junit4] 2> 501960 T1778 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 501960 T1778 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 501962 T1778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 501963 T1778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 501963 T1778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 501964 T1778 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 501964 T1778 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 501965 T1778 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 501965 T1778 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 501965 T1778 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 501966 T1778 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 501967 T1778 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 501970 T1778 oass.SolrIndexSearcher.<init> Opening Searcher@132ac70e main
[junit4:junit4] 2> 501970 T1778 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 501971 T1778 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 501975 T1779 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@132ac70e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 501977 T1778 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 501978 T1778 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53140/w/ai collection:collection1 shard:shard3
[junit4:junit4] 2> 501979 T1778 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 501984 T1778 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 501986 T1778 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 501986 T1778 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 501986 T1778 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53140/w/ai/collection1/
[junit4:junit4] 2> 501986 T1778 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 501986 T1778 oasc.SyncStrategy.syncToMe http://127.0.0.1:53140/w/ai/collection1/ has no replicas
[junit4:junit4] 2> 501987 T1778 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53140/w/ai/collection1/
[junit4:junit4] 2> 501987 T1778 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 502395 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 502402 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 502403 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 502403 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 502403 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 502403 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 502446 T1778 oasc.ZkController.register We are http://127.0.0.1:53140/w/ai/collection1/ and leader is http://127.0.0.1:53140/w/ai/collection1/
[junit4:junit4] 2> 502446 T1778 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53140/w/ai
[junit4:junit4] 2> 502446 T1778 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 502446 T1778 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 502447 T1778 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 502449 T1778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 502450 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 502451 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 502451 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 502533 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 502536 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60878
[junit4:junit4] 2> 502536 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 502537 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 502537 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645
[junit4:junit4] 2> 502537 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/solr.xml
[junit4:junit4] 2> 502538 T1701 oasc.CoreContainer.<init> New CoreContainer 1409131372
[junit4:junit4] 2> 502538 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/'
[junit4:junit4] 2> 502538 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/'
[junit4:junit4] 2> 502583 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 502583 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 502584 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 502584 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 502584 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 502584 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 502585 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 502585 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 502585 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 502585 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 502588 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 502589 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 502589 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 502589 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 502607 T1791 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@121cdd15 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 502608 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 502617 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 502620 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 502622 T1793 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f548999 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 502622 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 502626 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 503629 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60878_w%2Fai
[junit4:junit4] 2> 503630 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60878_w%2Fai
[junit4:junit4] 2> 503632 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 503632 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 503632 T1723 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 503633 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 503633 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 503633 T1730 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 503633 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 503633 T1793 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 503634 T1761 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 503635 T1745 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 503635 T1777 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 503639 T1794 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 503639 T1794 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 503908 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 503909 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53140_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53140/w/ai"}
[junit4:junit4] 2> 503913 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60878_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60878/w/ai"}
[junit4:junit4] 2> 503914 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 503914 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 503918 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 503918 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 503918 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 503918 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 503918 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 503918 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 504641 T1794 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1
[junit4:junit4] 2> 504641 T1794 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 504642 T1794 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 504642 T1794 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 504643 T1794 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1/'
[junit4:junit4] 2> 504643 T1794 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1/lib/README' to classloader
[junit4:junit4] 2> 504644 T1794 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 504667 T1794 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 504698 T1794 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 504700 T1794 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 504703 T1794 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 505001 T1794 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 505001 T1794 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 505002 T1794 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 505006 T1794 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 505008 T1794 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 505022 T1794 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 505024 T1794 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 505027 T1794 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 505028 T1794 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 505028 T1794 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 505029 T1794 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 505030 T1794 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 505030 T1794 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 505030 T1794 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 505030 T1794 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/
[junit4:junit4] 2> 505031 T1794 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
[junit4:junit4] 2> 505031 T1794 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 505031 T1794 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4
[junit4:junit4] 2> 505031 T1794 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/index/
[junit4:junit4] 2> 505032 T1794 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 505032 T1794 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/index
[junit4:junit4] 2> 505034 T1794 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 505035 T1794 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 505036 T1794 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 505036 T1794 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 505037 T1794 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 505037 T1794 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 505038 T1794 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 505038 T1794 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 505038 T1794 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 505038 T1794 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 505039 T1794 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 505040 T1794 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 505043 T1794 oass.SolrIndexSearcher.<init> Opening Searcher@7fda063 main
[junit4:junit4] 2> 505043 T1794 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 505043 T1794 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 505050 T1795 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7fda063 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 505051 T1794 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 505052 T1794 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60878/w/ai collection:collection1 shard:shard1
[junit4:junit4] 2> 505055 T1794 oasc.ZkController.register We are http://127.0.0.1:60878/w/ai/collection1/ and leader is http://127.0.0.1:50526/w/ai/collection1/
[junit4:junit4] 2> 505055 T1794 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60878/w/ai
[junit4:junit4] 2> 505055 T1794 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 505055 T1794 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1743 name=collection1 org.apache.solr.core.SolrCore@39a066cc url=http://127.0.0.1:60878/w/ai/collection1 node=127.0.0.1:60878_w%2Fai C1743_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:60878_w%2Fai, base_url=http://127.0.0.1:60878/w/ai}
[junit4:junit4] 2> 505055 T1796 C1743 P60878 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 505056 T1796 C1743 P60878 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 505056 T1794 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 505056 T1796 C1743 P60878 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 505057 T1796 C1743 P60878 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 505057 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 505058 T1796 C1743 P60878 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 505058 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 505058 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 505061 T1741 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 505144 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 505147 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56222
[junit4:junit4] 2> 505148 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 505148 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 505149 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252
[junit4:junit4] 2> 505149 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/solr.xml
[junit4:junit4] 2> 505149 T1701 oasc.CoreContainer.<init> New CoreContainer 1658125311
[junit4:junit4] 2> 505149 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/'
[junit4:junit4] 2> 505150 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/'
[junit4:junit4] 2> 505197 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 505197 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 505197 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 505198 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 505198 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 505198 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 505198 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 505199 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 505199 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 505199 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 505203 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 505203 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 505204 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 505204 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 505206 T1808 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1604bab6 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 505206 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 505208 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 505211 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 505213 T1810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23f74abf name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 505213 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 505216 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 505426 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 505427 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60878_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60878/w/ai"}
[junit4:junit4] 2> 505430 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 505430 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 505431 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 505430 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 505430 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 505431 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 505431 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 506062 T1741 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 506062 T1741 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:60878_w%252Fai&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 506219 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56222_w%2Fai
[junit4:junit4] 2> 506220 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56222_w%2Fai
[junit4:junit4] 2> 506223 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 506223 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 506223 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 506223 T1723 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 506223 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 506224 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506224 T1730 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 506224 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506224 T1810 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 506225 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506226 T1761 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 506226 T1745 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 506226 T1777 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 506226 T1793 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 506232 T1811 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 506232 T1811 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 506937 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 506938 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56222_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56222/w/ai"}
[junit4:junit4] 2> 506938 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 506938 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 506942 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506942 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506942 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506942 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506942 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506942 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 506942 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 507234 T1811 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1
[junit4:junit4] 2> 507235 T1811 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 507235 T1811 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 507235 T1811 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 507237 T1811 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1/'
[junit4:junit4] 2> 507237 T1811 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1/lib/README' to classloader
[junit4:junit4] 2> 507238 T1811 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 507270 T1811 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 507299 T1811 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 507300 T1811 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 507303 T1811 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 507623 T1811 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 507623 T1811 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 507624 T1811 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 507628 T1811 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 507630 T1811 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 507641 T1811 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 507644 T1811 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 507646 T1811 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 507647 T1811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 507647 T1811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 507647 T1811 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 507648 T1811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 507648 T1811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 507648 T1811 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 507649 T1811 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/
[junit4:junit4] 2> 507649 T1811 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
[junit4:junit4] 2> 507649 T1811 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 507649 T1811 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5
[junit4:junit4] 2> 507650 T1811 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/index/
[junit4:junit4] 2> 507650 T1811 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 507650 T1811 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/index
[junit4:junit4] 2> 507665 T1811 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 507665 T1811 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 507667 T1811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 507668 T1811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 507668 T1811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 507668 T1811 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 507669 T1811 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 507669 T1811 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 507669 T1811 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 507670 T1811 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 507670 T1811 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 507672 T1811 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 507674 T1811 oass.SolrIndexSearcher.<init> Opening Searcher@19cf35e0 main
[junit4:junit4] 2> 507675 T1811 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 507675 T1811 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 507679 T1812 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19cf35e0 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 507680 T1811 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 507680 T1811 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56222/w/ai collection:collection1 shard:shard2
[junit4:junit4] 2> 507683 T1811 oasc.ZkController.register We are http://127.0.0.1:56222/w/ai/collection1/ and leader is http://127.0.0.1:42132/w/ai/collection1/
[junit4:junit4] 2> 507683 T1811 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56222/w/ai
[junit4:junit4] 2> 507683 T1811 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 507684 T1811 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1744 name=collection1 org.apache.solr.core.SolrCore@733ee084 url=http://127.0.0.1:56222/w/ai/collection1 node=127.0.0.1:56222_w%2Fai C1744_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:56222_w%2Fai, base_url=http://127.0.0.1:56222/w/ai}
[junit4:junit4] 2> 507684 T1813 C1744 P56222 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 507684 T1813 C1744 P56222 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 507684 T1811 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 507685 T1813 C1744 P56222 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 507685 T1813 C1744 P56222 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 507685 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 507686 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 507686 T1813 C1744 P56222 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 507686 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 507690 T1755 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 507792 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 507794 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51169
[junit4:junit4] 2> 507795 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 507795 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 507796 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880
[junit4:junit4] 2> 507796 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/solr.xml
[junit4:junit4] 2> 507796 T1701 oasc.CoreContainer.<init> New CoreContainer 933683557
[junit4:junit4] 2> 507796 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/'
[junit4:junit4] 2> 507797 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/'
[junit4:junit4] 2> 507845 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 507846 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 507846 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 507846 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 507846 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 507847 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 507847 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 507847 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 507847 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 507848 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 507851 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 507851 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 507851 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 507852 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 507854 T1825 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6018acb8 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 507854 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 507855 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 507858 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 507860 T1827 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@299f380c name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 507860 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 507863 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1745 name=collection1 org.apache.solr.core.SolrCore@39a066cc url=http://127.0.0.1:60878/w/ai/collection1 node=127.0.0.1:60878_w%2Fai C1745_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:60878_w%2Fai, base_url=http://127.0.0.1:60878/w/ai}
[junit4:junit4] 2> 508063 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50526/w/ai/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 508064 T1796 C1745 P60878 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60878/w/ai START replicas=[http://127.0.0.1:50526/w/ai/collection1/] nUpdates=100
[junit4:junit4] 2> 508064 T1796 C1745 P60878 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 508065 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 508065 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 508065 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 508065 T1796 C1745 P60878 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 508065 T1796 C1745 P60878 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50526/w/ai/collection1/. core=collection1
[junit4:junit4] 2> 508065 T1796 C1745 P60878 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1746 name=collection1 org.apache.solr.core.SolrCore@5b7685c9 url=http://127.0.0.1:50526/w/ai/collection1 node=127.0.0.1:50526_w%2Fai C1746_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50526_w%2Fai, base_url=http://127.0.0.1:50526/w/ai, leader=true}
[junit4:junit4] 2> 508071 T1739 C1746 P50526 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 508072 T1736 C1746 P50526 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 508074 T1736 C1746 P50526 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 508074 T1736 C1746 P50526 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 508075 T1736 C1746 P50526 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 508075 T1736 C1746 P50526 oass.SolrIndexSearcher.<init> Opening Searcher@42600f06 realtime
[junit4:junit4] 2> 508075 T1736 C1746 P50526 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 508075 T1736 C1746 P50526 oasup.LogUpdateProcessor.finish [collection1] webapp=/w/ai path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4] 2> 508076 T1796 C1745 P60878 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 508077 T1796 C1745 P60878 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 508080 T1735 C1746 P50526 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 508081 T1735 C1746 P50526 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3
[junit4:junit4] 2> 508081 T1796 C1745 P60878 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 508081 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 508081 T1796 C1745 P60878 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 508081 T1796 C1745 P60878 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 508083 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 508452 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 508453 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56222_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56222/w/ai"}
[junit4:junit4] 2> 508458 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:60878_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:60878/w/ai"}
[junit4:junit4] 2> 508462 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508462 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508462 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508462 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508462 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508462 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508462 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508462 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508691 T1755 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 508691 T1755 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:56222_w%252Fai&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 508865 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51169_w%2Fai
[junit4:junit4] 2> 508866 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51169_w%2Fai
[junit4:junit4] 2> 508871 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508871 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508871 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508871 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508872 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 508872 T1810 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 508872 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 508872 T1730 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 508873 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 508873 T1723 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 508874 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 508874 T1793 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 508875 T1761 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 508875 T1777 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 508875 T1745 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 508875 T1827 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 508880 T1829 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 508881 T1829 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 509969 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 509970 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51169_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51169/w/ai"}
[junit4:junit4] 2> 509970 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 509970 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 509975 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 509975 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 509975 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 509975 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 509975 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 509975 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 509975 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 509975 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> ASYNC NEW_CORE C1747 name=collection1 org.apache.solr.core.SolrCore@733ee084 url=http://127.0.0.1:56222/w/ai/collection1 node=127.0.0.1:56222_w%2Fai C1747_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56222_w%2Fai, base_url=http://127.0.0.1:56222/w/ai}
[junit4:junit4] 2> 510692 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:42132/w/ai/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 510693 T1813 C1747 P56222 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56222/w/ai START replicas=[http://127.0.0.1:42132/w/ai/collection1/] nUpdates=100
[junit4:junit4] 2> 510693 T1813 C1747 P56222 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 510694 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 510694 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 510694 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 510694 T1813 C1747 P56222 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 510694 T1813 C1747 P56222 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:42132/w/ai/collection1/. core=collection1
[junit4:junit4] 2> 510694 T1813 C1747 P56222 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1748 name=collection1 org.apache.solr.core.SolrCore@5427339 url=http://127.0.0.1:42132/w/ai/collection1 node=127.0.0.1:42132_w%2Fai C1748_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:42132_w%2Fai, base_url=http://127.0.0.1:42132/w/ai, leader=true}
[junit4:junit4] 2> 510699 T1754 C1748 P42132 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 510701 T1757 C1748 P42132 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 510704 T1757 C1748 P42132 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 510705 T1757 C1748 P42132 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 510705 T1757 C1748 P42132 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 510705 T1757 C1748 P42132 oass.SolrIndexSearcher.<init> Opening Searcher@496b964 realtime
[junit4:junit4] 2> 510705 T1757 C1748 P42132 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 510706 T1757 C1748 P42132 oasup.LogUpdateProcessor.finish [collection1] webapp=/w/ai path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4] 2> 510707 T1813 C1747 P56222 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 510707 T1813 C1747 P56222 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 510708 T1755 C1748 P42132 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 510709 T1755 C1748 P42132 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 510709 T1813 C1747 P56222 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 510709 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 510709 T1813 C1747 P56222 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 510709 T1813 C1747 P56222 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 510711 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 510882 T1829 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1
[junit4:junit4] 2> 510882 T1829 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 510883 T1829 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 510883 T1829 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 510884 T1829 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1/'
[junit4:junit4] 2> 510885 T1829 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1/lib/README' to classloader
[junit4:junit4] 2> 510885 T1829 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 510912 T1829 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 510939 T1829 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 510940 T1829 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 510943 T1829 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 511253 T1829 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 511254 T1829 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 511254 T1829 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 511259 T1829 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 511261 T1829 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 511273 T1829 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 511276 T1829 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 511278 T1829 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 511279 T1829 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 511279 T1829 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 511279 T1829 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 511280 T1829 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 511280 T1829 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 511281 T1829 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 511281 T1829 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/
[junit4:junit4] 2> 511281 T1829 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
[junit4:junit4] 2> 511281 T1829 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 511282 T1829 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6
[junit4:junit4] 2> 511282 T1829 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/index/
[junit4:junit4] 2> 511282 T1829 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 511282 T1829 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/index
[junit4:junit4] 2> 511286 T1829 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 511286 T1829 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 511287 T1829 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 511288 T1829 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 511288 T1829 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 511288 T1829 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 511289 T1829 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 511289 T1829 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 511290 T1829 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 511290 T1829 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 511291 T1829 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 511292 T1829 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 511295 T1829 oass.SolrIndexSearcher.<init> Opening Searcher@7b2107c4 main
[junit4:junit4] 2> 511295 T1829 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 511295 T1829 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 511299 T1831 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b2107c4 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 511300 T1829 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 511301 T1829 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51169/w/ai collection:collection1 shard:shard3
[junit4:junit4] 2> 511304 T1829 oasc.ZkController.register We are http://127.0.0.1:51169/w/ai/collection1/ and leader is http://127.0.0.1:53140/w/ai/collection1/
[junit4:junit4] 2> 511304 T1829 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51169/w/ai
[junit4:junit4] 2> 511304 T1829 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 511304 T1829 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1749 name=collection1 org.apache.solr.core.SolrCore@708e4a05 url=http://127.0.0.1:51169/w/ai/collection1 node=127.0.0.1:51169_w%2Fai C1749_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:51169_w%2Fai, base_url=http://127.0.0.1:51169/w/ai}
[junit4:junit4] 2> 511305 T1832 C1749 P51169 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 511305 T1832 C1749 P51169 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 511305 T1829 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 511305 T1832 C1749 P51169 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 511306 T1832 C1749 P51169 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 511306 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 511307 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 511307 T1832 C1749 P51169 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 511307 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 511311 T1773 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 511386 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 511389 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47941
[junit4:junit4] 2> 511389 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 511389 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 511390 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501
[junit4:junit4] 2> 511390 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/solr.xml
[junit4:junit4] 2> 511390 T1701 oasc.CoreContainer.<init> New CoreContainer 273525130
[junit4:junit4] 2> 511391 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/'
[junit4:junit4] 2> 511391 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/'
[junit4:junit4] 2> 511447 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 511447 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 511447 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 511448 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 511448 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 511448 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 511448 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 511449 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 511449 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 511449 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 511452 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 511453 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 511453 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 511454 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 511455 T1844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e249d4b name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 511456 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 511457 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 511460 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 511461 T1846 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c25db54 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 511462 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 511465 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 511480 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 511481 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56222_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56222/w/ai"}
[junit4:junit4] 2> 511484 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51169_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51169/w/ai"}
[junit4:junit4] 2> 511487 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 511488 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 511488 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 511488 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 511488 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 511488 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 511488 T1846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 511488 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 511488 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 512311 T1773 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 512312 T1773 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:51169_w%252Fai&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 512468 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47941_w%2Fai
[junit4:junit4] 2> 512469 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47941_w%2Fai
[junit4:junit4] 2> 512471 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 512471 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 512471 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 512472 T1810 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512472 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 512472 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 512471 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 512472 T1723 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512473 T1730 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512473 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 512473 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 512473 T1846 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512474 T1846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 512474 T1827 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512475 T1793 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512475 T1777 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512476 T1761 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512476 T1745 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 512480 T1847 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 512480 T1847 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 512995 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 512996 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47941_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47941/w/ai"}
[junit4:junit4] 2> 512996 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 512996 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 513001 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513001 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513001 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513001 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513001 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513002 T1846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513001 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513001 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513001 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 513481 T1847 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1
[junit4:junit4] 2> 513482 T1847 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 513483 T1847 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 513483 T1847 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 513484 T1847 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1/'
[junit4:junit4] 2> 513485 T1847 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1/lib/README' to classloader
[junit4:junit4] 2> 513485 T1847 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 513525 T1847 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 513573 T1847 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 513575 T1847 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 513579 T1847 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 513932 T1847 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 513933 T1847 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4:junit4] 2> 513933 T1847 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 513938 T1847 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 513940 T1847 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 513951 T1847 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 513954 T1847 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 513957 T1847 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 513957 T1847 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 513958 T1847 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 513958 T1847 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 513975 T1847 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 513976 T1847 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 513976 T1847 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 513976 T1847 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/
[junit4:junit4] 2> 513976 T1847 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
[junit4:junit4] 2> 513977 T1847 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 513977 T1847 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7
[junit4:junit4] 2> 513977 T1847 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/index/
[junit4:junit4] 2> 513977 T1847 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 513978 T1847 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/index
[junit4:junit4] 2> 513980 T1847 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 513980 T1847 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 513982 T1847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 513982 T1847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 513983 T1847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 513983 T1847 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 513986 T1847 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 513989 T1847 oass.SolrIndexSearcher.<init> Opening Searcher@184a32e5 main
[junit4:junit4] 2> 513989 T1847 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 513989 T1847 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 513993 T1848 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@184a32e5 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 513994 T1847 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 513995 T1847 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47941/w/ai collection:collection1 shard:shard1
[junit4:junit4] 2> 513997 T1847 oasc.ZkController.register We are http://127.0.0.1:47941/w/ai/collection1/ and leader is http://127.0.0.1:50526/w/ai/collection1/
[junit4:junit4] 2> 513998 T1847 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47941/w/ai
[junit4:junit4] 2> 513998 T1847 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 513998 T1847 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C1750 name=collection1 org.apache.solr.core.SolrCore@36e06c36 url=http://127.0.0.1:47941/w/ai/collection1 node=127.0.0.1:47941_w%2Fai C1750_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47941_w%2Fai, base_url=http://127.0.0.1:47941/w/ai}
[junit4:junit4] 2> 513998 T1849 C1750 P47941 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 513999 T1849 C1750 P47941 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 513999 T1847 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 513999 T1849 C1750 P47941 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 513999 T1849 C1750 P47941 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 514000 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 514000 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 514000 T1849 C1750 P47941 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 514001 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 514005 T1736 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 8, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 514081 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 514084 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49852
[junit4:junit4] 2> 514084 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 514084 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 514084 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372650472195
[junit4:junit4] 2> 514085 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372650472195/solr.xml
[junit4:junit4] 2> 514085 T1701 oasc.CoreContainer.<init> New CoreContainer 224203437
[junit4:junit4] 2> 514085 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372650472195/'
[junit4:junit4] 2> 514085 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372650472195/'
[junit4:junit4] 2> 514122 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 514123 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 514123 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 514123 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 514123 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 514124 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 514124 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 514124 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 514124 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 514124 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 514127 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 514128 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
[junit4:junit4] 2> 514128 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 514129 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 514130 T1861 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@492434cd name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 514130 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 514131 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 514134 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 514136 T1863 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e122592 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 514136 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 514138 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C1751 name=collection1 org.apache.solr.core.SolrCore@708e4a05 url=http://127.0.0.1:51169/w/ai/collection1 node=127.0.0.1:51169_w%2Fai C1751_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:51169_w%2Fai, base_url=http://127.0.0.1:51169/w/ai}
[junit4:junit4] 2> 514312 T1832 C1751 P51169 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53140/w/ai/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 514313 T1832 C1751 P51169 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51169/w/ai START replicas=[http://127.0.0.1:53140/w/ai/collection1/] nUpdates=100
[junit4:junit4] 2> 514313 T1832 C1751 P51169 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 514313 T1832 C1751 P51169 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 514314 T1832 C1751 P51169 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 514314 T1832 C1751 P51169 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 514314 T1832 C1751 P51169 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 514314 T1832 C1751 P51169 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53140/w/ai/collection1/. core=collection1
[junit4:junit4] 2> 514314 T1832 C1751 P51169 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C1752 name=collection1 org.apache.solr.core.SolrCore@50febdb9 url=http://127.0.0.1:53140/w/ai/collection1 node=127.0.0.1:53140_w%2Fai C1752_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53140_w%2Fai, base_url=http://127.0.0.1:53140/w/ai, leader=true}
[junit4:junit4] 2> 514324 T1772 C1752 P53140 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 514324 T1771 C1752 P53140 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4] 2> 514327 T1772 C1752 P53140 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 514327 T1772 C1752 P53140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 514327 T1772 C1752 P53140 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4] 2> 514327 T1772 C1752 P53140 oass.SolrIndexSearcher.<init> Opening Searcher@35bfb8cd realtime
[junit4:junit4] 2> 514328 T1772 C1752 P53140 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 514328 T1772 C1752 P53140 oasup.LogUpdateProcessor.finish [collection1] webapp=/w/ai path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4] 2> ASYNC NEW_CORE C1753 name=collection1 org.apache.solr.core.SolrCore@708e4a05 url=http://127.0.0.1:51169/w/ai/collection1 node=127.0.0.1:51169_w%2Fai C1753_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:51169_w%2Fai, base_url=http://127.0.0.1:51169/w/ai}
[junit4:junit4] 2> 514329 T1832 C1753 P51169 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 514329 T1832 C1753 P51169 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 514330 T1770 C1752 P53140 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 514330 T1770 C1752 P53140 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 514331 T1832 C1753 P51169 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 514331 T1832 C1753 P51169 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 514331 T1832 C1753 P51169 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 514331 T1832 C1753 P51169 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 514332 T1832 C1753 P51169 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 514508 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 514509 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47941_w%2Fai",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47941/w/ai"}
[junit4:junit4] 2> 514512 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2>
[...truncated too long message...]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 60) Thread[id=1853, name=qtp446426214-1853 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
[junit4:junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 61) Thread[id=2005, name=Thread-858-SendThread(localhost.localdomain:40839), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
[junit4:junit4] 2> 62) Thread[id=1797, name=Thread-769, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Object.wait(Native Method)
[junit4:junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4:junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:1304)
[junit4:junit4] 2> 63) Thread[id=1891, name=qtp1023326616-1891 Acceptor0 SelectChannelConnector@127.0.0.1:57574, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 64) Thread[id=2010, name=Thread-886, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Object.wait(Native Method)
[junit4:junit4] 2> at java.lang.Object.wait(Object.java:502)
[junit4:junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:1304)
[junit4:junit4] 2> 65) Thread[id=1765, name=HashSessionScavenger-50, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Object.wait(Native Method)
[junit4:junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
[junit4:junit4] 2> at java.util.TimerThread.run(Timer.java:505)
[junit4:junit4] 2> 66) Thread[id=1925, name=qtp175335152-1925 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:85)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> 67) Thread[id=1995, name=qtp1422854206-1995 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
[junit4:junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=Lucene41(blocksize=128), text=Pulsing41(freqCutoff=14 minBlockSize=50 maxBlockSize=194), _version_=PostingsFormat(name=Memory doPackFST= true), rnd_b=Pulsing41(freqCutoff=6 minBlockSize=50 maxBlockSize=194), intDefault=PostingsFormat(name=Memory doPackFST= true), timestamp=Lucene41(blocksize=128), id=Pulsing41(freqCutoff=6 minBlockSize=50 maxBlockSize=194), a_t=Lucene41(blocksize=128), range_facet_sl=Pulsing41(freqCutoff=6 minBlockSize=50 maxBlockSize=194), range_facet_si=Lucene41(blocksize=128), other_tl1=PostingsFormat(name=Memory doPackFST= true), multiDefault=Lucene41(blocksize=128), a_si=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=de_AT, timezone=America/Santa_Isabel
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=81,free=43293928,total=139444224
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestPhraseSuggestions, SolrCoreTest, RecoveryZkTest, RAMDirectoryFactoryTest, PluginInfoTest, WordBreakSolrSpellCheckerTest, TestSolrQueryParserResource, ClusterStateUpdateTest, TestSweetSpotSimilarityFactory, CSVRequestHandlerTest, PrimUtilsTest, LoggingHandlerTest, TestSurroundQueryParser, DistributedTermsComponentTest, DirectUpdateHandlerOptimizeTest, TestFunctionQuery, TestQueryUtils, LeaderElectionTest, SynonymTokenizerTest, TermsComponentTest, FieldMutatingUpdateProcessorTest, TestXIncludeConfig, SimpleFacetsTest, TestSort, CircularListTest, TestSearchPerf, TestExtendedDismaxParser, QueryParsingTest, OverseerCollectionProcessorTest, RegexBoostProcessorTest, TestCopyFieldCollectionResource, TestCoreContainer, SolrTestCaseJ4Test, TestGroupingSearch, TestCharFilters, ZkCLITest, FastVectorHighlighterTest, TestManagedSchemaFieldResource, AutoCommitTest, TestSolrDeletionPolicy2, SolrInfoMBeanTest, SyncSliceTest, PreAnalyzedUpdateProcessorTest, TestSolrXml, TestSolrIndexConfig, SoftAutoCommitTest, TestFieldTypeCollectionResource, DocumentBuilderTest, ResponseLogComponentTest, TestRemoteStreaming, DirectSolrConnectionTest, TestSolrQueryParserDefaultOperatorResource, SolrIndexSplitterTest, TestPluginEnable, PeerSyncTest, ChaosMonkeyNothingIsSafeTest, TestJoin, TestPHPSerializedResponseWriter, CoreAdminHandlerTest, TestStressVersions, TestWriterPerf, SchemaVersionSpecificBehaviorTest, SliceStateTest, MBeansHandlerTest, DateFieldTest, DistributedQueryElevationComponentTest, CacheHeaderTest, TestDefaultSearchFieldResource, TestPropInjectDefaults, MinimalSchemaTest, LegacyHTMLStripCharFilterTest, BadComponentTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=67E9A14DA93DFB66 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_AT -Dtests.timezone=America/Santa_Isabel -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([67E9A14DA93DFB66]:0)
[junit4:junit4] Completed on J0 in 7225.29s, 1 test, 2 errors <<< FAILURES!
[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1248: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:892: There were test failures: 299 suites, 1290 tests, 1 suite-level error, 1 error, 15 ignored (6 assumptions)
Total time: 147 minutes 48 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b94 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.8.0-ea-b94) - Build # 6315 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
Looks like these ones are caused by the last commit in SOLR-4926 - the last commit point on the deletion policy can apparently be null.
I tried just getting the last commit point from the reader like before when the last commit point is null and that seemed to solve things. Not sure if there is a better fix or not though.
- Mark
On Jul 1, 2013, at 1:48 AM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6315/
> Java: 64bit/jdk1.8.0-ea-b94 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
>
> 2 tests failed.
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
>
> Error Message:
> Suite timeout exceeded (>= 7200000 msec).
>
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
> at __randomizedtesting.SeedInfo.seed([67E9A14DA93DFB66]:0)
>
>
> REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
>
> Error Message:
> Test abandoned because suite timeout was reached.
>
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
> at __randomizedtesting.SeedInfo.seed([67E9A14DA93DFB66]:0)
>
>
>
>
> Build Log:
> [...truncated 10293 lines...]
> [junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
> [junit4:junit4] 2> 489864 T3 oasc.CoreContainer.finalize ERROR CoreContainer was not shutdown prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!! instance=1040410215
> [junit4:junit4] 2> 489868 T1701 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /w/ai
> [junit4:junit4] 2> 489872 T1701 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
> [junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1372650448063
> [junit4:junit4] 2> 489872 T1701 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4:junit4] 2> 489873 T1702 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
> [junit4:junit4] 2> 489973 T1701 oasc.ZkTestServer.run start zk server on port:40839
> [junit4:junit4] 2> 489974 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 490095 T1708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@71521c3f name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 490095 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 490095 T1701 oascc.SolrZkClient.makePath makePath: /solr
> [junit4:junit4] 2> 490100 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 490101 T1710 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f95ec65 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 490101 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 490102 T1701 oascc.SolrZkClient.makePath makePath: /collections/collection1
> [junit4:junit4] 2> 490104 T1701 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
> [junit4:junit4] 2> 490106 T1701 oascc.SolrZkClient.makePath makePath: /collections/control_collection
> [junit4:junit4] 2> 490107 T1701 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
> [junit4:junit4] 2> 490109 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
> [junit4:junit4] 2> 490109 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
> [junit4:junit4] 2> 490112 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
> [junit4:junit4] 2> 490113 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
> [junit4:junit4] 2> 490115 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
> [junit4:junit4] 2> 490115 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
> [junit4:junit4] 2> 490117 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
> [junit4:junit4] 2> 490118 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
> [junit4:junit4] 2> 490119 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
> [junit4:junit4] 2> 490120 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
> [junit4:junit4] 2> 490122 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
> [junit4:junit4] 2> 490122 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
> [junit4:junit4] 2> 490124 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4:junit4] 2> 490124 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4:junit4] 2> 490126 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
> [junit4:junit4] 2> 490126 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
> [junit4:junit4] 2> 490128 T1701 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
> [junit4:junit4] 2> 490129 T1701 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
> [junit4:junit4] 2> 490194 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 490211 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42089
> [junit4:junit4] 2> 490211 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 490212 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 490212 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322
> [junit4:junit4] 2> 490212 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/solr.xml
> [junit4:junit4] 2> 490212 T1701 oasc.CoreContainer.<init> New CoreContainer 554851461
> [junit4:junit4] 2> 490213 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/'
> [junit4:junit4] 2> 490213 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/'
> [junit4:junit4] 2> 490252 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 490253 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 490253 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 490253 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 490253 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 490254 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 490254 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 490254 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 490254 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 490255 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 490257 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 490258 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 490258 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 490259 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 490261 T1721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e210446 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 490261 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 490263 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 490265 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 490267 T1723 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b4e59d5 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 490267 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 490268 T1701 oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4:junit4] 2> 490271 T1701 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
> [junit4:junit4] 2> 490273 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4:junit4] 2> 490274 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42089_w%2Fai
> [junit4:junit4] 2> 490275 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42089_w%2Fai
> [junit4:junit4] 2> 490277 T1701 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
> [junit4:junit4] 2> 490282 T1701 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
> [junit4:junit4] 2> 490284 T1701 oasc.Overseer.start Overseer (id=89958019764387843-127.0.0.1:42089_w%2Fai-n_0000000000) starting
> [junit4:junit4] 2> 490286 T1701 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
> [junit4:junit4] 2> 490289 T1725 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
> [junit4:junit4] 2> 490289 T1701 oascc.SolrZkClient.makePath makePath: /clusterstate.json
> [junit4:junit4] 2> 490291 T1701 oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4:junit4] 2> 490292 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 490294 T1724 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
> [junit4:junit4] 2> 490296 T1726 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 490296 T1726 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 491797 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 491798 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"1",
> [junit4:junit4] 2> "numShards":"1",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"control_collection",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:42089_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:42089/w/ai"}
> [junit4:junit4] 2> 491798 T1724 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
> [junit4:junit4] 2> 491798 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 491804 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 492297 T1726 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1
> [junit4:junit4] 2> 492298 T1726 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
> [junit4:junit4] 2> 492298 T1726 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 492299 T1726 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
> [junit4:junit4] 2> 492300 T1726 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1/'
> [junit4:junit4] 2> 492300 T1726 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1/lib/README' to classloader
> [junit4:junit4] 2> 492301 T1726 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 492332 T1726 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 492372 T1726 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 492374 T1726 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 492377 T1726 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 492707 T1726 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 492708 T1726 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 492709 T1726 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 492718 T1726 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 492721 T1726 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 492766 T1726 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 492771 T1726 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 492776 T1726 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 492778 T1726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 492778 T1726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 492779 T1726 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 492780 T1726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 492780 T1726 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 492780 T1726 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 492781 T1726 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372650448322/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/
> [junit4:junit4] 2> 492781 T1726 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
> [junit4:junit4] 2> 492781 T1726 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 492782 T1726 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data
> [junit4:junit4] 2> 492782 T1726 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/index/
> [junit4:junit4] 2> 492783 T1726 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 492783 T1726 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/index
> [junit4:junit4] 2> 492788 T1726 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/control/data/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 492788 T1726 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 492790 T1726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 492790 T1726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 492791 T1726 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 492792 T1726 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 492792 T1726 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 492792 T1726 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 492793 T1726 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 492793 T1726 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 492793 T1726 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 492796 T1726 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 492800 T1726 oass.SolrIndexSearcher.<init> Opening Searcher@4b4def74 main
> [junit4:junit4] 2> 492801 T1726 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 492801 T1726 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 492805 T1726 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 492806 T1726 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42089/w/ai collection:control_collection shard:shard1
> [junit4:junit4] 2> 492806 T1726 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
> [junit4:junit4] 2> 492807 T1727 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b4def74 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 492816 T1726 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 492819 T1726 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 492819 T1726 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 492819 T1726 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42089/w/ai/collection1/
> [junit4:junit4] 2> 492820 T1726 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 492820 T1726 oasc.SyncStrategy.syncToMe http://127.0.0.1:42089/w/ai/collection1/ has no replicas
> [junit4:junit4] 2> 492820 T1726 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42089/w/ai/collection1/
> [junit4:junit4] 2> 492820 T1726 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
> [junit4:junit4] 2> 493309 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 493319 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 493328 T1726 oasc.ZkController.register We are http://127.0.0.1:42089/w/ai/collection1/ and leader is http://127.0.0.1:42089/w/ai/collection1/
> [junit4:junit4] 2> 493329 T1726 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42089/w/ai
> [junit4:junit4] 2> 493329 T1726 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 493329 T1726 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 493329 T1726 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 493331 T1726 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 493333 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4] 2> 493333 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 493334 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 493338 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:
> [junit4:junit4] 2> 493339 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 493341 T1730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8c7606d name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 493342 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 493346 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 493352 T1701 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
> [junit4:junit4] 2> 493469 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 493484 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50526
> [junit4:junit4] 2> 493488 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 493488 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 493488 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543
> [junit4:junit4] 2> 493489 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/solr.xml
> [junit4:junit4] 2> 493489 T1701 oasc.CoreContainer.<init> New CoreContainer 1314004103
> [junit4:junit4] 2> 493490 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/'
> [junit4:junit4] 2> 493490 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/'
> [junit4:junit4] 2> 493541 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 493542 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 493542 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 493543 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 493543 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 493543 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 493544 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 493544 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 493544 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 493545 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 493548 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 493548 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 493549 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 493549 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 493551 T1743 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@74be1c42 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 493551 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 493553 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 493556 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 493558 T1745 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@767d49cc name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 493558 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 493568 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 494573 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50526_w%2Fai
> [junit4:junit4] 2> 494574 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50526_w%2Fai
> [junit4:junit4] 2> 494578 T1730 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 494578 T1723 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 494579 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 494579 T1745 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 494585 T1746 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 494585 T1746 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 494825 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 494826 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"1",
> [junit4:junit4] 2> "numShards":"1",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"control_collection",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:42089_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:42089/w/ai"}
> [junit4:junit4] 2> 494830 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"2",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:50526_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:50526/w/ai"}
> [junit4:junit4] 2> 494830 T1724 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
> [junit4:junit4] 2> 494831 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 494837 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 494837 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 494837 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 495588 T1746 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1
> [junit4:junit4] 2> 495588 T1746 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 495589 T1746 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 495589 T1746 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 495590 T1746 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1/'
> [junit4:junit4] 2> 495591 T1746 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1/lib/README' to classloader
> [junit4:junit4] 2> 495591 T1746 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 495619 T1746 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 495645 T1746 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 495647 T1746 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 495649 T1746 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 495945 T1746 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 495945 T1746 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 495946 T1746 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 495950 T1746 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 495952 T1746 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 495966 T1746 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 495970 T1746 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 495973 T1746 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 495974 T1746 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 495975 T1746 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 495975 T1746 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 495976 T1746 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 495976 T1746 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 495976 T1746 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 495976 T1746 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372650451543/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/
> [junit4:junit4] 2> 495977 T1746 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
> [junit4:junit4] 2> 495977 T1746 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 495977 T1746 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1
> [junit4:junit4] 2> 495977 T1746 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index/
> [junit4:junit4] 2> 495978 T1746 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 495978 T1746 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index
> [junit4:junit4] 2> 495981 T1746 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 495981 T1746 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 495982 T1746 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 495983 T1746 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 495983 T1746 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 495984 T1746 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 495984 T1746 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 495984 T1746 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 495984 T1746 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 495985 T1746 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 495985 T1746 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 495986 T1746 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 495989 T1746 oass.SolrIndexSearcher.<init> Opening Searcher@3d61bd6d main
> [junit4:junit4] 2> 495989 T1746 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 495990 T1746 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 495993 T1747 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3d61bd6d main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 495995 T1746 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 495995 T1746 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50526/w/ai collection:collection1 shard:shard1
> [junit4:junit4] 2> 495996 T1746 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
> [junit4:junit4] 2> 496001 T1746 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 496003 T1746 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 496003 T1746 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 496003 T1746 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50526/w/ai/collection1/
> [junit4:junit4] 2> 496003 T1746 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 496004 T1746 oasc.SyncStrategy.syncToMe http://127.0.0.1:50526/w/ai/collection1/ has no replicas
> [junit4:junit4] 2> 496004 T1746 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50526/w/ai/collection1/
> [junit4:junit4] 2> 496004 T1746 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
> [junit4:junit4] 2> 496344 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 496352 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 496352 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 496352 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 496359 T1746 oasc.ZkController.register We are http://127.0.0.1:50526/w/ai/collection1/ and leader is http://127.0.0.1:50526/w/ai/collection1/
> [junit4:junit4] 2> 496359 T1746 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50526/w/ai
> [junit4:junit4] 2> 496360 T1746 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 496360 T1746 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 496360 T1746 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 496361 T1746 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 496363 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4] 2> 496363 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 496363 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 496429 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 496431 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42132
> [junit4:junit4] 2> 496432 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 496432 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 496432 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557
> [junit4:junit4] 2> 496433 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/solr.xml
> [junit4:junit4] 2> 496433 T1701 oasc.CoreContainer.<init> New CoreContainer 732481326
> [junit4:junit4] 2> 496433 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/'
> [junit4:junit4] 2> 496433 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/'
> [junit4:junit4] 2> 496473 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 496474 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 496475 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 496475 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 496475 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 496475 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 496478 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 496479 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 496479 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 496479 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 496481 T1759 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c76b072 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 496481 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 496483 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 496485 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 496487 T1761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ccc4d8e name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 496487 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 496490 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 497494 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42132_w%2Fai
> [junit4:junit4] 2> 497495 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42132_w%2Fai
> [junit4:junit4] 2> 497497 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 497498 T1723 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 497498 T1730 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 497498 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 497498 T1761 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 497498 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 497499 T1745 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 497503 T1762 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 497503 T1762 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 497857 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 497858 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"2",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:50526_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:50526/w/ai"}
> [junit4:junit4] 2> 497862 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"3",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:42132_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:42132/w/ai"}
> [junit4:junit4] 2> 497862 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 497862 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4] 2> 497865 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 497865 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 497865 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 497865 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 498505 T1762 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1
> [junit4:junit4] 2> 498505 T1762 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 498506 T1762 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 498506 T1762 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 498507 T1762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1/'
> [junit4:junit4] 2> 498508 T1762 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1/lib/README' to classloader
> [junit4:junit4] 2> 498508 T1762 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 498530 T1762 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 498559 T1762 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 498561 T1762 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 498563 T1762 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 498844 T1762 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 498844 T1762 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 498845 T1762 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 498850 T1762 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 498852 T1762 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 498863 T1762 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 498866 T1762 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 498868 T1762 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 498869 T1762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 498869 T1762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 498869 T1762 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 498870 T1762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 498870 T1762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 498871 T1762 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 498871 T1762 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372650454557/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/
> [junit4:junit4] 2> 498871 T1762 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
> [junit4:junit4] 2> 498871 T1762 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 498872 T1762 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2
> [junit4:junit4] 2> 498872 T1762 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index/
> [junit4:junit4] 2> 498872 T1762 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 498872 T1762 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index
> [junit4:junit4] 2> 498875 T1762 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 498876 T1762 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 498877 T1762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 498878 T1762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 498878 T1762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 498878 T1762 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 498879 T1762 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 498879 T1762 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 498879 T1762 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 498879 T1762 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 498880 T1762 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 498881 T1762 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 498884 T1762 oass.SolrIndexSearcher.<init> Opening Searcher@1ffb2d75 main
> [junit4:junit4] 2> 498884 T1762 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 498884 T1762 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 498887 T1763 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ffb2d75 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 498889 T1762 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 498889 T1762 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42132/w/ai collection:collection1 shard:shard2
> [junit4:junit4] 2> 498890 T1762 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
> [junit4:junit4] 2> 498895 T1762 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 498897 T1762 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 498897 T1762 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 498897 T1762 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42132/w/ai/collection1/
> [junit4:junit4] 2> 498897 T1762 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 498898 T1762 oasc.SyncStrategy.syncToMe http://127.0.0.1:42132/w/ai/collection1/ has no replicas
> [junit4:junit4] 2> 498898 T1762 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42132/w/ai/collection1/
> [junit4:junit4] 2> 498898 T1762 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
> [junit4:junit4] 2> 499371 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 499377 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 499377 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 499377 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 499377 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 499403 T1762 oasc.ZkController.register We are http://127.0.0.1:42132/w/ai/collection1/ and leader is http://127.0.0.1:42132/w/ai/collection1/
> [junit4:junit4] 2> 499403 T1762 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42132/w/ai
> [junit4:junit4] 2> 499403 T1762 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 499403 T1762 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 499404 T1762 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 499412 T1762 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 499413 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4] 2> 499413 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 499413 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 499479 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 499487 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53140
> [junit4:junit4] 2> 499488 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 499488 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 499489 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607
> [junit4:junit4] 2> 499489 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/solr.xml
> [junit4:junit4] 2> 499489 T1701 oasc.CoreContainer.<init> New CoreContainer 620715449
> [junit4:junit4] 2> 499490 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/'
> [junit4:junit4] 2> 499490 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/'
> [junit4:junit4] 2> 499530 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 499530 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 499531 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 499532 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 499532 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 499532 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 499535 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 499536 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 499536 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 499536 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 499538 T1775 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ab80171 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 499538 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 499542 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 499546 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 499547 T1777 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d809383 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 499547 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 499551 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 500553 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53140_w%2Fai
> [junit4:junit4] 2> 500554 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53140_w%2Fai
> [junit4:junit4] 2> 500556 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 500556 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 500557 T1723 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 500557 T1777 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 500557 T1730 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 500557 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 500558 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 500559 T1745 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 500559 T1761 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 500563 T1778 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 500563 T1778 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 500882 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 500883 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"3",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:42132_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:42132/w/ai"}
> [junit4:junit4] 2> 500886 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"4",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:53140_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:53140/w/ai"}
> [junit4:junit4] 2> 500886 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 500886 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
> [junit4:junit4] 2> 500889 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 500889 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 500889 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 500889 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 500889 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 501564 T1778 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1
> [junit4:junit4] 2> 501564 T1778 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 501565 T1778 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 501565 T1778 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 501567 T1778 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1/'
> [junit4:junit4] 2> 501567 T1778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1/lib/README' to classloader
> [junit4:junit4] 2> 501567 T1778 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 501594 T1778 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 501620 T1778 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 501621 T1778 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 501624 T1778 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 501923 T1778 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 501924 T1778 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 501925 T1778 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 501929 T1778 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 501931 T1778 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 501944 T1778 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 501948 T1778 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 501951 T1778 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 501952 T1778 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 501953 T1778 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 501953 T1778 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 501954 T1778 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 501954 T1778 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 501954 T1778 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 501954 T1778 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372650457607/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/
> [junit4:junit4] 2> 501955 T1778 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
> [junit4:junit4] 2> 501955 T1778 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 501955 T1778 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3
> [junit4:junit4] 2> 501956 T1778 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index/
> [junit4:junit4] 2> 501956 T1778 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 501956 T1778 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index
> [junit4:junit4] 2> 501960 T1778 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 501960 T1778 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 501962 T1778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 501963 T1778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 501963 T1778 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 501964 T1778 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 501964 T1778 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 501965 T1778 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 501965 T1778 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 501965 T1778 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 501966 T1778 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 501967 T1778 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 501970 T1778 oass.SolrIndexSearcher.<init> Opening Searcher@132ac70e main
> [junit4:junit4] 2> 501970 T1778 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 501971 T1778 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 501975 T1779 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@132ac70e main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 501977 T1778 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 501978 T1778 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53140/w/ai collection:collection1 shard:shard3
> [junit4:junit4] 2> 501979 T1778 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
> [junit4:junit4] 2> 501984 T1778 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 501986 T1778 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 501986 T1778 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 501986 T1778 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53140/w/ai/collection1/
> [junit4:junit4] 2> 501986 T1778 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 501986 T1778 oasc.SyncStrategy.syncToMe http://127.0.0.1:53140/w/ai/collection1/ has no replicas
> [junit4:junit4] 2> 501987 T1778 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53140/w/ai/collection1/
> [junit4:junit4] 2> 501987 T1778 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
> [junit4:junit4] 2> 502395 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 502402 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 502403 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 502403 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 502403 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 502403 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 502446 T1778 oasc.ZkController.register We are http://127.0.0.1:53140/w/ai/collection1/ and leader is http://127.0.0.1:53140/w/ai/collection1/
> [junit4:junit4] 2> 502446 T1778 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53140/w/ai
> [junit4:junit4] 2> 502446 T1778 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 502446 T1778 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 502447 T1778 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 502449 T1778 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 502450 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4] 2> 502451 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 502451 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 502533 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 502536 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60878
> [junit4:junit4] 2> 502536 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 502537 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 502537 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645
> [junit4:junit4] 2> 502537 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/solr.xml
> [junit4:junit4] 2> 502538 T1701 oasc.CoreContainer.<init> New CoreContainer 1409131372
> [junit4:junit4] 2> 502538 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/'
> [junit4:junit4] 2> 502538 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/'
> [junit4:junit4] 2> 502583 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 502583 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 502584 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 502584 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 502584 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 502584 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 502585 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 502585 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 502585 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 502585 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 502588 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 502589 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 502589 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 502589 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 502607 T1791 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@121cdd15 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 502608 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 502617 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 502620 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 502622 T1793 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f548999 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 502622 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 502626 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 503629 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60878_w%2Fai
> [junit4:junit4] 2> 503630 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60878_w%2Fai
> [junit4:junit4] 2> 503632 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 503632 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 503632 T1723 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 503633 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 503633 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 503633 T1730 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 503633 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 503633 T1793 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 503634 T1761 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 503635 T1745 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 503635 T1777 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 503639 T1794 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 503639 T1794 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 503908 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 503909 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"4",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard3",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:53140_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:53140/w/ai"}
> [junit4:junit4] 2> 503913 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"5",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:60878_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:60878/w/ai"}
> [junit4:junit4] 2> 503914 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 503914 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 503918 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 503918 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 503918 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 503918 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 503918 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 503918 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 504641 T1794 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1
> [junit4:junit4] 2> 504641 T1794 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 504642 T1794 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 504642 T1794 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 504643 T1794 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1/'
> [junit4:junit4] 2> 504643 T1794 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1/lib/README' to classloader
> [junit4:junit4] 2> 504644 T1794 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 504667 T1794 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 504698 T1794 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 504700 T1794 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 504703 T1794 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 505001 T1794 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 505001 T1794 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 505002 T1794 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 505006 T1794 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 505008 T1794 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 505022 T1794 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 505024 T1794 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 505027 T1794 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 505028 T1794 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 505028 T1794 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 505029 T1794 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 505030 T1794 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 505030 T1794 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 505030 T1794 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 505030 T1794 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372650460645/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/
> [junit4:junit4] 2> 505031 T1794 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
> [junit4:junit4] 2> 505031 T1794 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 505031 T1794 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4
> [junit4:junit4] 2> 505031 T1794 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/index/
> [junit4:junit4] 2> 505032 T1794 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 505032 T1794 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/index
> [junit4:junit4] 2> 505034 T1794 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty4/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 505035 T1794 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 505036 T1794 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 505036 T1794 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 505037 T1794 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 505037 T1794 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 505038 T1794 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 505038 T1794 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 505038 T1794 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 505038 T1794 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 505039 T1794 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 505040 T1794 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 505043 T1794 oass.SolrIndexSearcher.<init> Opening Searcher@7fda063 main
> [junit4:junit4] 2> 505043 T1794 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 505043 T1794 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 505050 T1795 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7fda063 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 505051 T1794 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 505052 T1794 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60878/w/ai collection:collection1 shard:shard1
> [junit4:junit4] 2> 505055 T1794 oasc.ZkController.register We are http://127.0.0.1:60878/w/ai/collection1/ and leader is http://127.0.0.1:50526/w/ai/collection1/
> [junit4:junit4] 2> 505055 T1794 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60878/w/ai
> [junit4:junit4] 2> 505055 T1794 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 505055 T1794 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C1743 name=collection1 org.apache.solr.core.SolrCore@39a066cc url=http://127.0.0.1:60878/w/ai/collection1 node=127.0.0.1:60878_w%2Fai C1743_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:60878_w%2Fai, base_url=http://127.0.0.1:60878/w/ai}
> [junit4:junit4] 2> 505055 T1796 C1743 P60878 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 505056 T1796 C1743 P60878 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 505056 T1794 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 505056 T1796 C1743 P60878 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 505057 T1796 C1743 P60878 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 505057 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4] 2> 505058 T1796 C1743 P60878 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 505058 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 505058 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 505061 T1741 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 505144 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 505147 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56222
> [junit4:junit4] 2> 505148 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 505148 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 505149 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252
> [junit4:junit4] 2> 505149 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/solr.xml
> [junit4:junit4] 2> 505149 T1701 oasc.CoreContainer.<init> New CoreContainer 1658125311
> [junit4:junit4] 2> 505149 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/'
> [junit4:junit4] 2> 505150 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/'
> [junit4:junit4] 2> 505197 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 505197 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 505197 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 505198 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 505198 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 505198 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 505198 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 505199 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 505199 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 505199 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 505203 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 505203 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 505204 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 505204 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 505206 T1808 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1604bab6 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 505206 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 505208 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 505211 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 505213 T1810 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23f74abf name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 505213 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 505216 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 505426 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 505427 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"5",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:60878_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:60878/w/ai"}
> [junit4:junit4] 2> 505430 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 505430 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 505431 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 505430 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 505430 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 505431 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 505431 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 506062 T1741 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4] 2> 506062 T1741 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:60878_w%252Fai&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
> [junit4:junit4] 2> 506219 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56222_w%2Fai
> [junit4:junit4] 2> 506220 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56222_w%2Fai
> [junit4:junit4] 2> 506223 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 506223 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 506223 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 506223 T1723 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 506223 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 506224 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506224 T1730 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 506224 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506224 T1810 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 506225 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506226 T1761 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 506226 T1745 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 506226 T1777 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 506226 T1793 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 506232 T1811 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 506232 T1811 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 506937 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 506938 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"6",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:56222_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:56222/w/ai"}
> [junit4:junit4] 2> 506938 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 506938 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4] 2> 506942 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506942 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506942 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506942 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506942 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506942 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 506942 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 507234 T1811 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1
> [junit4:junit4] 2> 507235 T1811 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 507235 T1811 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 507235 T1811 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 507237 T1811 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1/'
> [junit4:junit4] 2> 507237 T1811 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1/lib/README' to classloader
> [junit4:junit4] 2> 507238 T1811 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 507270 T1811 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 507299 T1811 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 507300 T1811 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 507303 T1811 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 507623 T1811 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 507623 T1811 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 507624 T1811 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 507628 T1811 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 507630 T1811 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 507641 T1811 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 507644 T1811 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 507646 T1811 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 507647 T1811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 507647 T1811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 507647 T1811 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 507648 T1811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 507648 T1811 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 507648 T1811 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 507649 T1811 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372650463252/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/
> [junit4:junit4] 2> 507649 T1811 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
> [junit4:junit4] 2> 507649 T1811 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 507649 T1811 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5
> [junit4:junit4] 2> 507650 T1811 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/index/
> [junit4:junit4] 2> 507650 T1811 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 507650 T1811 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/index
> [junit4:junit4] 2> 507665 T1811 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty5/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 507665 T1811 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 507667 T1811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 507668 T1811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 507668 T1811 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 507668 T1811 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 507669 T1811 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 507669 T1811 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 507669 T1811 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 507670 T1811 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 507670 T1811 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 507672 T1811 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 507674 T1811 oass.SolrIndexSearcher.<init> Opening Searcher@19cf35e0 main
> [junit4:junit4] 2> 507675 T1811 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 507675 T1811 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 507679 T1812 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19cf35e0 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 507680 T1811 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 507680 T1811 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56222/w/ai collection:collection1 shard:shard2
> [junit4:junit4] 2> 507683 T1811 oasc.ZkController.register We are http://127.0.0.1:56222/w/ai/collection1/ and leader is http://127.0.0.1:42132/w/ai/collection1/
> [junit4:junit4] 2> 507683 T1811 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56222/w/ai
> [junit4:junit4] 2> 507683 T1811 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 507684 T1811 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C1744 name=collection1 org.apache.solr.core.SolrCore@733ee084 url=http://127.0.0.1:56222/w/ai/collection1 node=127.0.0.1:56222_w%2Fai C1744_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:56222_w%2Fai, base_url=http://127.0.0.1:56222/w/ai}
> [junit4:junit4] 2> 507684 T1813 C1744 P56222 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 507684 T1813 C1744 P56222 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 507684 T1811 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 507685 T1813 C1744 P56222 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 507685 T1813 C1744 P56222 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 507685 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4] 2> 507686 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 507686 T1813 C1744 P56222 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 507686 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 507690 T1755 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 507792 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 507794 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51169
> [junit4:junit4] 2> 507795 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 507795 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 507796 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880
> [junit4:junit4] 2> 507796 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/solr.xml
> [junit4:junit4] 2> 507796 T1701 oasc.CoreContainer.<init> New CoreContainer 933683557
> [junit4:junit4] 2> 507796 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/'
> [junit4:junit4] 2> 507797 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/'
> [junit4:junit4] 2> 507845 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 507846 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 507846 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 507846 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 507846 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 507847 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 507847 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 507847 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 507847 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 507848 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 507851 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 507851 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 507851 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 507852 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 507854 T1825 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6018acb8 name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 507854 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 507855 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 507858 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 507860 T1827 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@299f380c name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 507860 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 507863 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> ASYNC NEW_CORE C1745 name=collection1 org.apache.solr.core.SolrCore@39a066cc url=http://127.0.0.1:60878/w/ai/collection1 node=127.0.0.1:60878_w%2Fai C1745_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:60878_w%2Fai, base_url=http://127.0.0.1:60878/w/ai}
> [junit4:junit4] 2> 508063 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50526/w/ai/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4] 2> 508064 T1796 C1745 P60878 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60878/w/ai START replicas=[http://127.0.0.1:50526/w/ai/collection1/] nUpdates=100
> [junit4:junit4] 2> 508064 T1796 C1745 P60878 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4] 2> 508065 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4] 2> 508065 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4] 2> 508065 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4] 2> 508065 T1796 C1745 P60878 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4] 2> 508065 T1796 C1745 P60878 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50526/w/ai/collection1/. core=collection1
> [junit4:junit4] 2> 508065 T1796 C1745 P60878 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> ASYNC NEW_CORE C1746 name=collection1 org.apache.solr.core.SolrCore@5b7685c9 url=http://127.0.0.1:50526/w/ai/collection1 node=127.0.0.1:50526_w%2Fai C1746_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50526_w%2Fai, base_url=http://127.0.0.1:50526/w/ai, leader=true}
> [junit4:junit4] 2> 508071 T1739 C1746 P50526 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
> [junit4:junit4] 2> 508072 T1736 C1746 P50526 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4] 2> 508074 T1736 C1746 P50526 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 508074 T1736 C1746 P50526 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 508075 T1736 C1746 P50526 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
> [junit4:junit4] 2> 508075 T1736 C1746 P50526 oass.SolrIndexSearcher.<init> Opening Searcher@42600f06 realtime
> [junit4:junit4] 2> 508075 T1736 C1746 P50526 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4] 2> 508075 T1736 C1746 P50526 oasup.LogUpdateProcessor.finish [collection1] webapp=/w/ai path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
> [junit4:junit4] 2> 508076 T1796 C1745 P60878 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 508077 T1796 C1745 P60878 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4] 2> 508080 T1735 C1746 P50526 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 508081 T1735 C1746 P50526 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=3
> [junit4:junit4] 2> 508081 T1796 C1745 P60878 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4] 2> 508081 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4] 2> 508081 T1796 C1745 P60878 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 508081 T1796 C1745 P60878 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 508083 T1796 C1745 P60878 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4] 2> 508452 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 508453 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"6",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:56222_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:56222/w/ai"}
> [junit4:junit4] 2> 508458 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"5",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:60878_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:60878/w/ai"}
> [junit4:junit4] 2> 508462 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508462 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508462 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508462 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508462 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508462 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508462 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508462 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508691 T1755 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4] 2> 508691 T1755 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:56222_w%252Fai&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
> [junit4:junit4] 2> 508865 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51169_w%2Fai
> [junit4:junit4] 2> 508866 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51169_w%2Fai
> [junit4:junit4] 2> 508871 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508871 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508871 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508871 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508872 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 508872 T1810 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 508872 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 508872 T1730 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 508873 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 508873 T1723 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 508874 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 508874 T1793 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 508875 T1761 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 508875 T1777 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 508875 T1745 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 508875 T1827 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 508880 T1829 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 508881 T1829 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 509969 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 509970 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"7",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:51169_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:51169/w/ai"}
> [junit4:junit4] 2> 509970 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 509970 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
> [junit4:junit4] 2> 509975 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 509975 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 509975 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 509975 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 509975 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 509975 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 509975 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 509975 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> ASYNC NEW_CORE C1747 name=collection1 org.apache.solr.core.SolrCore@733ee084 url=http://127.0.0.1:56222/w/ai/collection1 node=127.0.0.1:56222_w%2Fai C1747_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:56222_w%2Fai, base_url=http://127.0.0.1:56222/w/ai}
> [junit4:junit4] 2> 510692 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:42132/w/ai/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4] 2> 510693 T1813 C1747 P56222 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56222/w/ai START replicas=[http://127.0.0.1:42132/w/ai/collection1/] nUpdates=100
> [junit4:junit4] 2> 510693 T1813 C1747 P56222 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4] 2> 510694 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4] 2> 510694 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4] 2> 510694 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4] 2> 510694 T1813 C1747 P56222 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4] 2> 510694 T1813 C1747 P56222 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:42132/w/ai/collection1/. core=collection1
> [junit4:junit4] 2> 510694 T1813 C1747 P56222 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> ASYNC NEW_CORE C1748 name=collection1 org.apache.solr.core.SolrCore@5427339 url=http://127.0.0.1:42132/w/ai/collection1 node=127.0.0.1:42132_w%2Fai C1748_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:42132_w%2Fai, base_url=http://127.0.0.1:42132/w/ai, leader=true}
> [junit4:junit4] 2> 510699 T1754 C1748 P42132 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
> [junit4:junit4] 2> 510701 T1757 C1748 P42132 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4] 2> 510704 T1757 C1748 P42132 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 510705 T1757 C1748 P42132 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 510705 T1757 C1748 P42132 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
> [junit4:junit4] 2> 510705 T1757 C1748 P42132 oass.SolrIndexSearcher.<init> Opening Searcher@496b964 realtime
> [junit4:junit4] 2> 510705 T1757 C1748 P42132 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4] 2> 510706 T1757 C1748 P42132 oasup.LogUpdateProcessor.finish [collection1] webapp=/w/ai path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
> [junit4:junit4] 2> 510707 T1813 C1747 P56222 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 510707 T1813 C1747 P56222 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4] 2> 510708 T1755 C1748 P42132 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 510709 T1755 C1748 P42132 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
> [junit4:junit4] 2> 510709 T1813 C1747 P56222 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4] 2> 510709 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4] 2> 510709 T1813 C1747 P56222 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 510709 T1813 C1747 P56222 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 510711 T1813 C1747 P56222 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4] 2> 510882 T1829 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1
> [junit4:junit4] 2> 510882 T1829 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 510883 T1829 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 510883 T1829 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 510884 T1829 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1/'
> [junit4:junit4] 2> 510885 T1829 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1/lib/README' to classloader
> [junit4:junit4] 2> 510885 T1829 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 510912 T1829 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 510939 T1829 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 510940 T1829 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 510943 T1829 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 511253 T1829 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 511254 T1829 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 511254 T1829 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 511259 T1829 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 511261 T1829 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 511273 T1829 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 511276 T1829 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 511278 T1829 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 511279 T1829 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 511279 T1829 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 511279 T1829 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 511280 T1829 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 511280 T1829 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 511281 T1829 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 511281 T1829 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372650465880/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/
> [junit4:junit4] 2> 511281 T1829 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
> [junit4:junit4] 2> 511281 T1829 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 511282 T1829 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6
> [junit4:junit4] 2> 511282 T1829 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/index/
> [junit4:junit4] 2> 511282 T1829 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 511282 T1829 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/index
> [junit4:junit4] 2> 511286 T1829 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty6/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 511286 T1829 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 511287 T1829 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 511288 T1829 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 511288 T1829 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 511288 T1829 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 511289 T1829 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 511289 T1829 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 511290 T1829 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 511290 T1829 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 511291 T1829 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 511292 T1829 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 511295 T1829 oass.SolrIndexSearcher.<init> Opening Searcher@7b2107c4 main
> [junit4:junit4] 2> 511295 T1829 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 511295 T1829 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 511299 T1831 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b2107c4 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 511300 T1829 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 511301 T1829 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51169/w/ai collection:collection1 shard:shard3
> [junit4:junit4] 2> 511304 T1829 oasc.ZkController.register We are http://127.0.0.1:51169/w/ai/collection1/ and leader is http://127.0.0.1:53140/w/ai/collection1/
> [junit4:junit4] 2> 511304 T1829 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51169/w/ai
> [junit4:junit4] 2> 511304 T1829 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 511304 T1829 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C1749 name=collection1 org.apache.solr.core.SolrCore@708e4a05 url=http://127.0.0.1:51169/w/ai/collection1 node=127.0.0.1:51169_w%2Fai C1749_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:51169_w%2Fai, base_url=http://127.0.0.1:51169/w/ai}
> [junit4:junit4] 2> 511305 T1832 C1749 P51169 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 511305 T1832 C1749 P51169 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 511305 T1829 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 511305 T1832 C1749 P51169 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 511306 T1832 C1749 P51169 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 511306 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4] 2> 511307 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 511307 T1832 C1749 P51169 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 511307 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 511311 T1773 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 511386 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 511389 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47941
> [junit4:junit4] 2> 511389 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 511389 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 511390 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501
> [junit4:junit4] 2> 511390 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/solr.xml
> [junit4:junit4] 2> 511390 T1701 oasc.CoreContainer.<init> New CoreContainer 273525130
> [junit4:junit4] 2> 511391 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/'
> [junit4:junit4] 2> 511391 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/'
> [junit4:junit4] 2> 511447 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 511447 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 511447 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 511448 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 511448 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 511448 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 511448 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 511449 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 511449 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 511449 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 511452 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 511453 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 511453 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 511454 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 511455 T1844 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e249d4b name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 511456 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 511457 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 511460 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 511461 T1846 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c25db54 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 511462 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 511465 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 511480 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 511481 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"6",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:56222_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:56222/w/ai"}
> [junit4:junit4] 2> 511484 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"7",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard3",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:51169_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:51169/w/ai"}
> [junit4:junit4] 2> 511487 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 511488 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 511488 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 511488 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 511488 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 511488 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 511488 T1846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 511488 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 511488 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 512311 T1773 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4] 2> 512312 T1773 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:51169_w%252Fai&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
> [junit4:junit4] 2> 512468 T1701 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47941_w%2Fai
> [junit4:junit4] 2> 512469 T1701 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47941_w%2Fai
> [junit4:junit4] 2> 512471 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 512471 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 512471 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 512472 T1810 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512472 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 512472 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 512471 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 512472 T1723 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512473 T1730 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512473 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 512473 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 512473 T1846 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512474 T1846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 512474 T1827 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512475 T1793 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512475 T1777 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512476 T1761 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512476 T1745 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 512480 T1847 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 512480 T1847 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 512995 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 512996 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"8",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:47941_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:47941/w/ai"}
> [junit4:junit4] 2> 512996 T1724 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 512996 T1724 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 513001 T1827 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513001 T1793 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513001 T1745 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513001 T1761 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513001 T1810 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513002 T1846 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513001 T1723 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513001 T1730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513001 T1777 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 513481 T1847 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1
> [junit4:junit4] 2> 513482 T1847 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 513483 T1847 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 513483 T1847 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 513484 T1847 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1/'
> [junit4:junit4] 2> 513485 T1847 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1/lib/README' to classloader
> [junit4:junit4] 2> 513485 T1847 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 513525 T1847 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 513573 T1847 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 513575 T1847 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 513579 T1847 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 513932 T1847 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 513933 T1847 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
> [junit4:junit4] 2> 513933 T1847 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 513938 T1847 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 513940 T1847 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 513951 T1847 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 513954 T1847 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 513957 T1847 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 513957 T1847 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 513958 T1847 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 513958 T1847 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 513975 T1847 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 513976 T1847 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 513976 T1847 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 513976 T1847 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372650469501/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/
> [junit4:junit4] 2> 513976 T1847 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@298ef72d
> [junit4:junit4] 2> 513977 T1847 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 513977 T1847 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7
> [junit4:junit4] 2> 513977 T1847 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/index/
> [junit4:junit4] 2> 513977 T1847 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 513978 T1847 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/index
> [junit4:junit4] 2> 513980 T1847 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty7/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 513980 T1847 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 513982 T1847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 513982 T1847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 513983 T1847 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 513983 T1847 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 513984 T1847 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 513986 T1847 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 513989 T1847 oass.SolrIndexSearcher.<init> Opening Searcher@184a32e5 main
> [junit4:junit4] 2> 513989 T1847 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 513989 T1847 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 513993 T1848 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@184a32e5 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 513994 T1847 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 513995 T1847 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47941/w/ai collection:collection1 shard:shard1
> [junit4:junit4] 2> 513997 T1847 oasc.ZkController.register We are http://127.0.0.1:47941/w/ai/collection1/ and leader is http://127.0.0.1:50526/w/ai/collection1/
> [junit4:junit4] 2> 513998 T1847 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47941/w/ai
> [junit4:junit4] 2> 513998 T1847 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 513998 T1847 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C1750 name=collection1 org.apache.solr.core.SolrCore@36e06c36 url=http://127.0.0.1:47941/w/ai/collection1 node=127.0.0.1:47941_w%2Fai C1750_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47941_w%2Fai, base_url=http://127.0.0.1:47941/w/ai}
> [junit4:junit4] 2> 513998 T1849 C1750 P47941 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 513999 T1849 C1750 P47941 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 513999 T1847 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 513999 T1849 C1750 P47941 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 513999 T1849 C1750 P47941 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 514000 T1701 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
> [junit4:junit4] 2> 514000 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 514000 T1849 C1750 P47941 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 514001 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 514005 T1736 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 8, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 514081 T1701 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 514084 T1701 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49852
> [junit4:junit4] 2> 514084 T1701 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 514084 T1701 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 514084 T1701 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372650472195
> [junit4:junit4] 2> 514085 T1701 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372650472195/solr.xml
> [junit4:junit4] 2> 514085 T1701 oasc.CoreContainer.<init> New CoreContainer 224203437
> [junit4:junit4] 2> 514085 T1701 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372650472195/'
> [junit4:junit4] 2> 514085 T1701 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372650472195/'
> [junit4:junit4] 2> 514122 T1701 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 514123 T1701 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 514123 T1701 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 514123 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 514123 T1701 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 514124 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 514124 T1701 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 514124 T1701 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 514124 T1701 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 514124 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 514127 T1701 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 514128 T1701 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:40839/solr
> [junit4:junit4] 2> 514128 T1701 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 514129 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 514130 T1861 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@492434cd name:ZooKeeperConnection Watcher:127.0.0.1:40839 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 514130 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 514131 T1701 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 514134 T1701 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 514136 T1863 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e122592 name:ZooKeeperConnection Watcher:127.0.0.1:40839/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 514136 T1701 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 514138 T1701 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> ASYNC NEW_CORE C1751 name=collection1 org.apache.solr.core.SolrCore@708e4a05 url=http://127.0.0.1:51169/w/ai/collection1 node=127.0.0.1:51169_w%2Fai C1751_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:51169_w%2Fai, base_url=http://127.0.0.1:51169/w/ai}
> [junit4:junit4] 2> 514312 T1832 C1751 P51169 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:53140/w/ai/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4] 2> 514313 T1832 C1751 P51169 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51169/w/ai START replicas=[http://127.0.0.1:53140/w/ai/collection1/] nUpdates=100
> [junit4:junit4] 2> 514313 T1832 C1751 P51169 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4] 2> 514313 T1832 C1751 P51169 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4] 2> 514314 T1832 C1751 P51169 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4] 2> 514314 T1832 C1751 P51169 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4] 2> 514314 T1832 C1751 P51169 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4] 2> 514314 T1832 C1751 P51169 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:53140/w/ai/collection1/. core=collection1
> [junit4:junit4] 2> 514314 T1832 C1751 P51169 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> ASYNC NEW_CORE C1752 name=collection1 org.apache.solr.core.SolrCore@50febdb9 url=http://127.0.0.1:53140/w/ai/collection1 node=127.0.0.1:53140_w%2Fai C1752_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:53140_w%2Fai, base_url=http://127.0.0.1:53140/w/ai, leader=true}
> [junit4:junit4] 2> 514324 T1772 C1752 P53140 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4] 2> 514324 T1771 C1752 P53140 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
> [junit4:junit4] 2> 514327 T1772 C1752 P53140 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372650448063/jetty3/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 514327 T1772 C1752 P53140 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 514327 T1772 C1752 P53140 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
> [junit4:junit4] 2> 514327 T1772 C1752 P53140 oass.SolrIndexSearcher.<init> Opening Searcher@35bfb8cd realtime
> [junit4:junit4] 2> 514328 T1772 C1752 P53140 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4] 2> 514328 T1772 C1752 P53140 oasup.LogUpdateProcessor.finish [collection1] webapp=/w/ai path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
> [junit4:junit4] 2> ASYNC NEW_CORE C1753 name=collection1 org.apache.solr.core.SolrCore@708e4a05 url=http://127.0.0.1:51169/w/ai/collection1 node=127.0.0.1:51169_w%2Fai C1753_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:51169_w%2Fai, base_url=http://127.0.0.1:51169/w/ai}
> [junit4:junit4] 2> 514329 T1832 C1753 P51169 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 514329 T1832 C1753 P51169 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4] 2> 514330 T1770 C1752 P53140 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 514330 T1770 C1752 P53140 oasc.SolrCore.execute [collection1] webapp=/w/ai path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
> [junit4:junit4] 2> 514331 T1832 C1753 P51169 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4] 2> 514331 T1832 C1753 P51169 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4] 2> 514331 T1832 C1753 P51169 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 514331 T1832 C1753 P51169 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 514332 T1832 C1753 P51169 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4] 2> 514508 T1724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 514509 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"8",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:47941_w%2Fai",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:47941/w/ai"}
> [junit4:junit4] 2> 514512 T1724 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"7",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard3",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2>
>
> [...truncated too long message...]
>
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
> [junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
> [junit4:junit4] 2> 60) Thread[id=1853, name=qtp446426214-1853 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> [junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
> [junit4:junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> [junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> [junit4:junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
> [junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
> [junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> [junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> [junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
> [junit4:junit4] 2> 61) Thread[id=2005, name=Thread-858-SendThread(localhost.localdomain:40839), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> at java.lang.Thread.sleep(Native Method)
> [junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
> [junit4:junit4] 2> 62) Thread[id=1797, name=Thread-769, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
> [junit4:junit4] 2> at java.lang.Object.wait(Object.java:502)
> [junit4:junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:1304)
> [junit4:junit4] 2> 63) Thread[id=1891, name=qtp1023326616-1891 Acceptor0 SelectChannelConnector@127.0.0.1:57574, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
> [junit4:junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
> [junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> [junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
> [junit4:junit4] 2> 64) Thread[id=2010, name=Thread-886, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
> [junit4:junit4] 2> at java.lang.Object.wait(Object.java:502)
> [junit4:junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:1304)
> [junit4:junit4] 2> 65) Thread[id=1765, name=HashSessionScavenger-50, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> at java.lang.Object.wait(Native Method)
> [junit4:junit4] 2> at java.util.TimerThread.mainLoop(Timer.java:552)
> [junit4:junit4] 2> at java.util.TimerThread.run(Timer.java:505)
> [junit4:junit4] 2> 66) Thread[id=1925, name=qtp175335152-1925 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:85)
> [junit4:junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
> [junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
> [junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> [junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> [junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
> [junit4:junit4] 2> 67) Thread[id=1995, name=qtp1422854206-1995 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> [junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:257)
> [junit4:junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> [junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> [junit4:junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> [junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
> [junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
> [junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> [junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:724)
> [junit4:junit4] 2> NOTE: test params are: codec=Lucene42: {range_facet_l=Lucene41(blocksize=128), text=Pulsing41(freqCutoff=14 minBlockSize=50 maxBlockSize=194), _version_=PostingsFormat(name=Memory doPackFST= true), rnd_b=Pulsing41(freqCutoff=6 minBlockSize=50 maxBlockSize=194), intDefault=PostingsFormat(name=Memory doPackFST= true), timestamp=Lucene41(blocksize=128), id=Pulsing41(freqCutoff=6 minBlockSize=50 maxBlockSize=194), a_t=Lucene41(blocksize=128), range_facet_sl=Pulsing41(freqCutoff=6 minBlockSize=50 maxBlockSize=194), range_facet_si=Lucene41(blocksize=128), other_tl1=PostingsFormat(name=Memory doPackFST= true), multiDefault=Lucene41(blocksize=128), a_si=PostingsFormat(name=Memory doPackFST= true)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=de_AT, timezone=America/Santa_Isabel
> [junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=81,free=43293928,total=139444224
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestPhraseSuggestions, SolrCoreTest, RecoveryZkTest, RAMDirectoryFactoryTest, PluginInfoTest, WordBreakSolrSpellCheckerTest, TestSolrQueryParserResource, ClusterStateUpdateTest, TestSweetSpotSimilarityFactory, CSVRequestHandlerTest, PrimUtilsTest, LoggingHandlerTest, TestSurroundQueryParser, DistributedTermsComponentTest, DirectUpdateHandlerOptimizeTest, TestFunctionQuery, TestQueryUtils, LeaderElectionTest, SynonymTokenizerTest, TermsComponentTest, FieldMutatingUpdateProcessorTest, TestXIncludeConfig, SimpleFacetsTest, TestSort, CircularListTest, TestSearchPerf, TestExtendedDismaxParser, QueryParsingTest, OverseerCollectionProcessorTest, RegexBoostProcessorTest, TestCopyFieldCollectionResource, TestCoreContainer, SolrTestCaseJ4Test, TestGroupingSearch, TestCharFilters, ZkCLITest, FastVectorHighlighterTest, TestManagedSchemaFieldResource, AutoCommitTest, TestSolrDeletionPolicy2, SolrInfoMBeanTest, SyncSliceTest, PreAnalyzedUpdateProcessorTest, TestSolrXml, TestSolrIndexConfig, SoftAutoCommitTest, TestFieldTypeCollectionResource, DocumentBuilderTest, ResponseLogComponentTest, TestRemoteStreaming, DirectSolrConnectionTest, TestSolrQueryParserDefaultOperatorResource, SolrIndexSplitterTest, TestPluginEnable, PeerSyncTest, ChaosMonkeyNothingIsSafeTest, TestJoin, TestPHPSerializedResponseWriter, CoreAdminHandlerTest, TestStressVersions, TestWriterPerf, SchemaVersionSpecificBehaviorTest, SliceStateTest, MBeansHandlerTest, DateFieldTest, DistributedQueryElevationComponentTest, CacheHeaderTest, TestDefaultSearchFieldResource, TestPropInjectDefaults, MinimalSchemaTest, LegacyHTMLStripCharFilterTest, BadComponentTest, ChaosMonkeySafeLeaderTest]
> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=67E9A14DA93DFB66 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=de_AT -Dtests.timezone=America/Santa_Isabel -Dtests.file.encoding=UTF-8
> [junit4:junit4] ERROR 0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
> [junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([67E9A14DA93DFB66]:0)
> [junit4:junit4] Completed on J0 in 7225.29s, 1 test, 2 errors <<< FAILURES!
>
> [...truncated 11 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1248: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:892: There were test failures: 299 suites, 1290 tests, 1 suite-level error, 1 error, 15 ignored (6 assumptions)
>
> Total time: 147 minutes 48 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.8.0-ea-b94 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org