You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/16 03:03:18 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_21) - Build # 6085
- Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6085/
Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([93165A8081EFE724]:0)
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([93165A8081EFE724]:0)
Build Log:
[...truncated 10298 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 439597 T2314 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 439601 T2314 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1371337343383
[junit4:junit4] 2> 439602 T2314 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 439602 T2315 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 439702 T2314 oasc.ZkTestServer.run start zk server on port:39914
[junit4:junit4] 2> 439704 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 439784 T2321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@519ad4d2 name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 439784 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 439785 T2314 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 439791 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 439793 T2323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bbcb568 name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 439794 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 439794 T2314 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 439798 T2314 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 439802 T2314 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 439805 T2314 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 439808 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 439809 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 439814 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 439815 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 439819 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 439820 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 439823 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 439824 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 439827 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 439828 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 439832 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 439832 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 439836 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 439837 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 439840 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 439841 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 439844 T2314 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 439845 T2314 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 439929 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 439973 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38682
[junit4:junit4] 2> 439974 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 439975 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 439975 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630
[junit4:junit4] 2> 439976 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630/solr.xml
[junit4:junit4] 2> 439976 T2314 oasc.CoreContainer.<init> New CoreContainer 253535963
[junit4:junit4] 2> 439977 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630/'
[junit4:junit4] 2> 439977 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630/'
[junit4:junit4] 2> 440020 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 440021 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 440021 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 440021 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 440022 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 440022 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 440022 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 440023 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 440023 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 440023 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 440026 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 440027 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 440027 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 440028 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 440029 T2334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4df58885 name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 440030 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 440031 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 440033 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 440034 T2336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@612bc85e name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 440034 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 440035 T2314 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 440038 T2314 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 440039 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 440041 T2314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38682_
[junit4:junit4] 2> 440041 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38682_
[junit4:junit4] 2> 440043 T2314 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 440047 T2314 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 440048 T2314 oasc.Overseer.start Overseer (id=89871964136079363-127.0.0.1:38682_-n_0000000000) starting
[junit4:junit4] 2> 440050 T2314 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 440052 T2338 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 440052 T2314 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 440053 T2314 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 440054 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 440056 T2337 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 440057 T2339 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 440058 T2339 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 441559 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 441560 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38682_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38682"}
[junit4:junit4] 2> 441561 T2337 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 441561 T2337 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 441566 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 442059 T2339 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630/collection1
[junit4:junit4] 2> 442059 T2339 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 442060 T2339 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 442060 T2339 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 442062 T2339 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630/collection1/'
[junit4:junit4] 2> 442063 T2339 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630/collection1/lib/README' to classloader
[junit4:junit4] 2> 442064 T2339 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 442093 T2339 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 442124 T2339 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 442126 T2339 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 442129 T2339 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 442459 T2339 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 442464 T2339 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 442466 T2339 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 442480 T2339 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 442484 T2339 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 442488 T2339 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 442489 T2339 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 442489 T2339 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 442489 T2339 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 442490 T2339 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 442490 T2339 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 442491 T2339 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 442491 T2339 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371337343630/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/control/data/
[junit4:junit4] 2> 442491 T2339 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@443c86bc
[junit4:junit4] 2> 442492 T2339 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 442493 T2339 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/control/data
[junit4:junit4] 2> 442493 T2339 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/control/data/index/
[junit4:junit4] 2> 442493 T2339 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 442494 T2339 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/control/data/index
[junit4:junit4] 2> 442497 T2339 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 442498 T2339 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 442499 T2339 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 442499 T2339 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 442500 T2339 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 442500 T2339 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 442500 T2339 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 442500 T2339 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 442501 T2339 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 442501 T2339 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 442501 T2339 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 442503 T2339 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 442506 T2339 oass.SolrIndexSearcher.<init> Opening Searcher@708e475d main
[junit4:junit4] 2> 442506 T2339 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 442507 T2339 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 442510 T2340 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@708e475d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 442510 T2339 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 442510 T2339 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38682 collection:control_collection shard:shard1
[junit4:junit4] 2> 442511 T2339 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 442515 T2339 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 442517 T2339 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 442517 T2339 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 442517 T2339 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38682/collection1/
[junit4:junit4] 2> 442517 T2339 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 442517 T2339 oasc.SyncStrategy.syncToMe http://127.0.0.1:38682/collection1/ has no replicas
[junit4:junit4] 2> 442518 T2339 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38682/collection1/
[junit4:junit4] 2> 442518 T2339 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 443071 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 443082 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 443122 T2339 oasc.ZkController.register We are http://127.0.0.1:38682/collection1/ and leader is http://127.0.0.1:38682/collection1/
[junit4:junit4] 2> 443123 T2339 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38682
[junit4:junit4] 2> 443123 T2339 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 443123 T2339 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 443124 T2339 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 443126 T2339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 443128 T2314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 443129 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 443129 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 443136 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 443137 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443140 T2343 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b243ba4 name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443141 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443142 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 443145 T2314 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 443228 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 443230 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52519
[junit4:junit4] 2> 443231 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 443231 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 443232 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927
[junit4:junit4] 2> 443232 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927/solr.xml
[junit4:junit4] 2> 443233 T2314 oasc.CoreContainer.<init> New CoreContainer 1417599168
[junit4:junit4] 2> 443233 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927/'
[junit4:junit4] 2> 443234 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927/'
[junit4:junit4] 2> 443280 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 443280 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 443281 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 443281 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 443281 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 443281 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 443282 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 443282 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 443282 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 443283 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 443286 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 443286 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 443287 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 443287 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443289 T2354 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15838c8b name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443289 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443291 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 443293 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 443294 T2356 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18c02297 name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 443295 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 443297 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 444300 T2314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52519_
[junit4:junit4] 2> 444302 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52519_
[junit4:junit4] 2> 444305 T2343 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 444305 T2356 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 444305 T2336 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 444306 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 444312 T2357 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 444313 T2357 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 444589 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 444590 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:38682_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:38682"}
[junit4:junit4] 2> 444595 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52519_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52519"}
[junit4:junit4] 2> 444595 T2337 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 444596 T2337 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 444601 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 444601 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 444601 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 445314 T2357 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927/collection1
[junit4:junit4] 2> 445315 T2357 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 445315 T2357 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 445315 T2357 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 445316 T2357 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927/collection1/'
[junit4:junit4] 2> 445317 T2357 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927/collection1/lib/README' to classloader
[junit4:junit4] 2> 445317 T2357 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 445346 T2357 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 445372 T2357 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 445373 T2357 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 445376 T2357 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 445701 T2357 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 445705 T2357 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 445707 T2357 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 445716 T2357 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 445718 T2357 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 445721 T2357 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 445722 T2357 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 445722 T2357 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 445722 T2357 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 445723 T2357 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 445723 T2357 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 445723 T2357 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 445724 T2357 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371337346927/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1/
[junit4:junit4] 2> 445724 T2357 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@443c86bc
[junit4:junit4] 2> 445724 T2357 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 445725 T2357 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1
[junit4:junit4] 2> 445725 T2357 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1/index/
[junit4:junit4] 2> 445725 T2357 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 445726 T2357 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1/index
[junit4:junit4] 2> 445729 T2357 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 445729 T2357 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 445731 T2357 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 445732 T2357 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 445733 T2357 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 445733 T2357 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 445734 T2357 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 445734 T2357 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 445735 T2357 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 445736 T2357 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 445736 T2357 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 445740 T2357 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 445745 T2357 oass.SolrIndexSearcher.<init> Opening Searcher@c8e3a88 main
[junit4:junit4] 2> 445746 T2357 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 445746 T2357 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 445751 T2358 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c8e3a88 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 445753 T2357 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 445754 T2357 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52519 collection:collection1 shard:shard1
[junit4:junit4] 2> 445755 T2357 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 445763 T2357 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 445766 T2357 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 445767 T2357 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 445767 T2357 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52519/collection1/
[junit4:junit4] 2> 445768 T2357 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 445768 T2357 oasc.SyncStrategy.syncToMe http://127.0.0.1:52519/collection1/ has no replicas
[junit4:junit4] 2> 445769 T2357 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52519/collection1/
[junit4:junit4] 2> 445769 T2357 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 446108 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 446118 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 446118 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 446118 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 446127 T2357 oasc.ZkController.register We are http://127.0.0.1:52519/collection1/ and leader is http://127.0.0.1:52519/collection1/
[junit4:junit4] 2> 446127 T2357 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52519
[junit4:junit4] 2> 446128 T2357 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 446128 T2357 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 446128 T2357 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 446130 T2357 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 446132 T2314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 446132 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 446133 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 446229 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 446232 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46325
[junit4:junit4] 2> 446232 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 446233 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 446233 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920
[junit4:junit4] 2> 446233 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920/solr.xml
[junit4:junit4] 2> 446234 T2314 oasc.CoreContainer.<init> New CoreContainer 1842274950
[junit4:junit4] 2> 446234 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920/'
[junit4:junit4] 2> 446234 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920/'
[junit4:junit4] 2> 446275 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 446276 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 446276 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 446276 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 446277 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 446277 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 446277 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 446278 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 446278 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 446278 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 446281 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 446282 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 446282 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 446283 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446284 T2370 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66c49344 name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446285 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446287 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 446289 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 446290 T2372 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f69350d name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 446291 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 446294 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 447297 T2314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46325_
[junit4:junit4] 2> 447299 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46325_
[junit4:junit4] 2> 447302 T2356 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 447302 T2343 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 447303 T2336 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 447303 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 447303 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 447302 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 447303 T2372 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 447310 T2373 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 447310 T2373 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 447627 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 447628 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:52519_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:52519"}
[junit4:junit4] 2> 447633 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46325_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46325"}
[junit4:junit4] 2> 447634 T2337 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 447634 T2337 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 447639 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 447640 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 447640 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 447639 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 448312 T2373 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920/collection1
[junit4:junit4] 2> 448313 T2373 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 448314 T2373 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 448314 T2373 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 448316 T2373 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920/collection1/'
[junit4:junit4] 2> 448317 T2373 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920/collection1/lib/README' to classloader
[junit4:junit4] 2> 448317 T2373 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 448358 T2373 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 448387 T2373 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 448389 T2373 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 448392 T2373 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 448705 T2373 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 448709 T2373 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 448712 T2373 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 448721 T2373 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 448724 T2373 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 448726 T2373 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 448727 T2373 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 448727 T2373 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 448727 T2373 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 448728 T2373 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 448728 T2373 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 448728 T2373 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 448728 T2373 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371337349920/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2/
[junit4:junit4] 2> 448729 T2373 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@443c86bc
[junit4:junit4] 2> 448729 T2373 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 448729 T2373 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2
[junit4:junit4] 2> 448730 T2373 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2/index/
[junit4:junit4] 2> 448730 T2373 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 448730 T2373 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2/index
[junit4:junit4] 2> 448733 T2373 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 448733 T2373 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 448734 T2373 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 448735 T2373 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 448735 T2373 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 448735 T2373 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 448736 T2373 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 448740 T2373 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 448741 T2373 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 448741 T2373 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 448741 T2373 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 448744 T2373 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 448746 T2373 oass.SolrIndexSearcher.<init> Opening Searcher@51936cdb main
[junit4:junit4] 2> 448747 T2373 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 448747 T2373 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 448751 T2373 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 448751 T2373 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46325 collection:collection1 shard:shard2
[junit4:junit4] 2> 448752 T2374 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51936cdb main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 448752 T2373 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 448756 T2373 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 448758 T2373 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 448758 T2373 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 448758 T2373 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:46325/collection1/
[junit4:junit4] 2> 448758 T2373 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 448759 T2373 oasc.SyncStrategy.syncToMe http://127.0.0.1:46325/collection1/ has no replicas
[junit4:junit4] 2> 448759 T2373 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:46325/collection1/
[junit4:junit4] 2> 448759 T2373 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 449155 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 449165 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 449165 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 449165 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 449165 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 449214 T2373 oasc.ZkController.register We are http://127.0.0.1:46325/collection1/ and leader is http://127.0.0.1:46325/collection1/
[junit4:junit4] 2> 449214 T2373 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46325
[junit4:junit4] 2> 449215 T2373 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 449215 T2373 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 449215 T2373 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 449218 T2373 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 449220 T2314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 449221 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 449222 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 449300 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 449302 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:56083
[junit4:junit4] 2> 449302 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 449303 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 449303 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009
[junit4:junit4] 2> 449303 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009/solr.xml
[junit4:junit4] 2> 449304 T2314 oasc.CoreContainer.<init> New CoreContainer 1902820536
[junit4:junit4] 2> 449304 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009/'
[junit4:junit4] 2> 449305 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009/'
[junit4:junit4] 2> 449345 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 449346 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 449346 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 449346 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 449347 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 449353 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 449353 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 449354 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 449354 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 449354 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 449357 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 449358 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 449358 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 449359 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 449360 T2386 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bc8f74e name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 449361 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 449362 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 449365 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 449366 T2388 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11a7cfb9 name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 449366 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 449374 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 450377 T2314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56083_
[junit4:junit4] 2> 450379 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:56083_
[junit4:junit4] 2> 450382 T2372 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 450383 T2356 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 450383 T2343 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 450383 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450383 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450383 T2388 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 450383 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450383 T2336 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 450385 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450391 T2389 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 450392 T2389 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 450673 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 450674 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:46325_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:46325"}
[junit4:junit4] 2> 450678 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56083_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56083"}
[junit4:junit4] 2> 450679 T2337 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 450679 T2337 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 450684 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450684 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450684 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450684 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 450684 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 451393 T2389 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009/collection1
[junit4:junit4] 2> 451394 T2389 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 451398 T2389 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 451398 T2389 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 451399 T2389 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009/collection1/'
[junit4:junit4] 2> 451400 T2389 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009/collection1/lib/README' to classloader
[junit4:junit4] 2> 451400 T2389 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 451422 T2389 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 451455 T2389 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 451457 T2389 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 451460 T2389 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 451788 T2389 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 451793 T2389 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 451795 T2389 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 451804 T2389 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 451807 T2389 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 451809 T2389 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 451810 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 451810 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 451811 T2389 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 451811 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 451811 T2389 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 451812 T2389 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 451812 T2389 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371337353009/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty3/
[junit4:junit4] 2> 451812 T2389 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@443c86bc
[junit4:junit4] 2> 451813 T2389 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 451813 T2389 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty3
[junit4:junit4] 2> 451813 T2389 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty3/index/
[junit4:junit4] 2> 451814 T2389 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 451814 T2389 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty3/index
[junit4:junit4] 2> 451823 T2389 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 451823 T2389 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 451826 T2389 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 451826 T2389 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 451827 T2389 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 451828 T2389 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 451829 T2389 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 451829 T2389 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 451830 T2389 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 451830 T2389 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 451831 T2389 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 451835 T2389 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 451840 T2389 oass.SolrIndexSearcher.<init> Opening Searcher@5103e2ce main
[junit4:junit4] 2> 451841 T2389 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 451841 T2389 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 451846 T2390 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5103e2ce main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 451848 T2389 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 451848 T2389 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:56083 collection:collection1 shard:shard3
[junit4:junit4] 2> 451849 T2389 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 451858 T2389 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 451861 T2389 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 451861 T2389 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 451862 T2389 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:56083/collection1/
[junit4:junit4] 2> 451862 T2389 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 451863 T2389 oasc.SyncStrategy.syncToMe http://127.0.0.1:56083/collection1/ has no replicas
[junit4:junit4] 2> 451863 T2389 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:56083/collection1/
[junit4:junit4] 2> 451863 T2389 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 452192 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 452201 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 452202 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 452201 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 452201 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 452201 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 452220 T2389 oasc.ZkController.register We are http://127.0.0.1:56083/collection1/ and leader is http://127.0.0.1:56083/collection1/
[junit4:junit4] 2> 452221 T2389 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56083
[junit4:junit4] 2> 452221 T2389 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 452221 T2389 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 452221 T2389 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 452224 T2389 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 452226 T2314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 452226 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 452227 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 452308 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 452310 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:51322
[junit4:junit4] 2> 452310 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 452311 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 452311 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014
[junit4:junit4] 2> 452311 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014/solr.xml
[junit4:junit4] 2> 452312 T2314 oasc.CoreContainer.<init> New CoreContainer 32140698
[junit4:junit4] 2> 452312 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014/'
[junit4:junit4] 2> 452312 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014/'
[junit4:junit4] 2> 452357 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 452357 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 452357 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 452358 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 452358 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 452358 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 452359 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 452359 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 452359 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 452360 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 452363 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 452363 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 452363 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 452364 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 452365 T2402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f1a8dda name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 452366 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 452367 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 452370 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 452371 T2404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@577843d4 name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 452371 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 452373 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 453375 T2314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51322_
[junit4:junit4] 2> 453377 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51322_
[junit4:junit4] 2> 453381 T2372 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 453381 T2356 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 453381 T2404 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 453381 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453381 T2343 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 453381 T2388 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 453382 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453381 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453381 T2336 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 453383 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453384 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453397 T2405 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 453398 T2405 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 453709 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 453710 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:56083_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:56083"}
[junit4:junit4] 2> 453714 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51322_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51322"}
[junit4:junit4] 2> 453714 T2337 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 453715 T2337 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 453719 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453719 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453719 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453719 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453719 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 453719 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 454400 T2405 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014/collection1
[junit4:junit4] 2> 454401 T2405 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 454402 T2405 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 454402 T2405 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 454405 T2405 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014/collection1/'
[junit4:junit4] 2> 454406 T2405 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014/collection1/lib/README' to classloader
[junit4:junit4] 2> 454406 T2405 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 454434 T2405 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 454469 T2405 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 454470 T2405 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 454473 T2405 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 454830 T2405 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 454836 T2405 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 454840 T2405 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 454858 T2405 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 454862 T2405 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 454868 T2405 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 454870 T2405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 454870 T2405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 454871 T2405 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 454872 T2405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 454873 T2405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 454873 T2405 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 454873 T2405 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371337356014/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/
[junit4:junit4] 2> 454874 T2405 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@443c86bc
[junit4:junit4] 2> 454874 T2405 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 454875 T2405 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4
[junit4:junit4] 2> 454876 T2405 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index/
[junit4:junit4] 2> 454876 T2405 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 454876 T2405 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index
[junit4:junit4] 2> 454879 T2405 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 454879 T2405 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 454882 T2405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 454882 T2405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 454883 T2405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 454884 T2405 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 454884 T2405 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 454884 T2405 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 454885 T2405 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 454885 T2405 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 454886 T2405 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 454889 T2405 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 454894 T2405 oass.SolrIndexSearcher.<init> Opening Searcher@500dfd5b main
[junit4:junit4] 2> 454895 T2405 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 454895 T2405 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 454904 T2406 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@500dfd5b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 454906 T2405 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 454906 T2405 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51322 collection:collection1 shard:shard1
[junit4:junit4] 2> 454909 T2405 oasc.ZkController.register We are http://127.0.0.1:51322/collection1/ and leader is http://127.0.0.1:52519/collection1/
[junit4:junit4] 2> 454909 T2405 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:51322
[junit4:junit4] 2> 454910 T2405 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 454910 T2405 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2530 name=collection1 org.apache.solr.core.SolrCore@15683790 url=http://127.0.0.1:51322/collection1 node=127.0.0.1:51322_ C2530_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:51322_, base_url=http://127.0.0.1:51322}
[junit4:junit4] 2> 454910 T2407 C2530 P51322 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 454911 T2405 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 454912 T2407 C2530 P51322 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 454912 T2407 C2530 P51322 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 454912 T2407 C2530 P51322 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 454912 T2314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 454913 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 454914 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 454914 T2407 C2530 P51322 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 454922 T2352 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 455014 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 455016 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:32871
[junit4:junit4] 2> 455017 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 455017 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 455018 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700
[junit4:junit4] 2> 455018 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700/solr.xml
[junit4:junit4] 2> 455018 T2314 oasc.CoreContainer.<init> New CoreContainer 979359743
[junit4:junit4] 2> 455019 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700/'
[junit4:junit4] 2> 455019 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700/'
[junit4:junit4] 2> 455076 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 455077 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 455078 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 455078 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 455078 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 455079 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 455079 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 455080 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 455080 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 455081 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 455086 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 455087 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 455087 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 455088 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 455093 T2419 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e0f023c name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 455093 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 455095 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 455102 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 455103 T2421 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@199a2044 name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 455103 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 455106 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 455233 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 455234 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51322_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51322"}
[junit4:junit4] 2> 455237 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 455237 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 455238 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 455237 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 455237 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 455237 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 455237 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 455923 T2352 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 455924 T2352 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:51322_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4] 2> 456110 T2314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:32871_
[junit4:junit4] 2> 456112 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:32871_
[junit4:junit4] 2> 456116 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 456116 T2343 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 456117 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456117 T2421 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 456116 T2388 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 456116 T2336 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 456118 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456118 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456118 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456117 T2372 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 456120 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456117 T2356 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 456119 T2404 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 456121 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456125 T2422 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 456125 T2422 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 456744 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 456745 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32871_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32871"}
[junit4:junit4] 2> 456745 T2337 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 456746 T2337 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 456774 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456774 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456774 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456774 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456774 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456774 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 456774 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 457127 T2422 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700/collection1
[junit4:junit4] 2> 457127 T2422 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 457128 T2422 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 457129 T2422 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 457131 T2422 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700/collection1/'
[junit4:junit4] 2> 457131 T2422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700/collection1/lib/README' to classloader
[junit4:junit4] 2> 457132 T2422 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 457162 T2422 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 457188 T2422 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 457190 T2422 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 457193 T2422 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 457508 T2422 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 457512 T2422 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 457514 T2422 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 457527 T2422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 457530 T2422 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 457533 T2422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 457534 T2422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 457534 T2422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 457534 T2422 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 457535 T2422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 457536 T2422 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 457536 T2422 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 457536 T2422 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371337358700/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/
[junit4:junit4] 2> 457536 T2422 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@443c86bc
[junit4:junit4] 2> 457537 T2422 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 457537 T2422 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5
[junit4:junit4] 2> 457537 T2422 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index/
[junit4:junit4] 2> 457538 T2422 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 457538 T2422 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index
[junit4:junit4] 2> 457541 T2422 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 457541 T2422 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 457543 T2422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 457544 T2422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 457544 T2422 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 457545 T2422 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 457545 T2422 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 457545 T2422 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 457546 T2422 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 457546 T2422 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 457546 T2422 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 457550 T2422 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 457553 T2422 oass.SolrIndexSearcher.<init> Opening Searcher@610f9025 main
[junit4:junit4] 2> 457554 T2422 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 457554 T2422 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 457559 T2423 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@610f9025 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 457561 T2422 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 457561 T2422 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:32871 collection:collection1 shard:shard2
[junit4:junit4] 2> 457564 T2422 oasc.ZkController.register We are http://127.0.0.1:32871/collection1/ and leader is http://127.0.0.1:46325/collection1/
[junit4:junit4] 2> 457564 T2422 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:32871
[junit4:junit4] 2> 457565 T2422 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 457565 T2422 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2531 name=collection1 org.apache.solr.core.SolrCore@5bb83605 url=http://127.0.0.1:32871/collection1 node=127.0.0.1:32871_ C2531_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:32871_, base_url=http://127.0.0.1:32871}
[junit4:junit4] 2> 457566 T2424 C2531 P32871 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 457566 T2424 C2531 P32871 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 457566 T2422 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 457567 T2424 C2531 P32871 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 457567 T2424 C2531 P32871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 457568 T2314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 457568 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 457568 T2424 C2531 P32871 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 457569 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 457577 T2366 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 457648 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 457651 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55840
[junit4:junit4] 2> 457652 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 457652 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 457653 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356
[junit4:junit4] 2> 457654 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356/solr.xml
[junit4:junit4] 2> 457654 T2314 oasc.CoreContainer.<init> New CoreContainer 1368144230
[junit4:junit4] 2> 457655 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356/'
[junit4:junit4] 2> 457656 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356/'
[junit4:junit4] 2> 457697 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 457697 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 457697 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 457698 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 457698 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 457698 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 457699 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 457699 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 457699 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 457699 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 457703 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 457703 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 457703 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 457704 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 457706 T2436 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@52578137 name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 457706 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 457707 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 457710 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 457711 T2438 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19fb6953 name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 457711 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 457714 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C2532 name=collection1 org.apache.solr.core.SolrCore@15683790 url=http://127.0.0.1:51322/collection1 node=127.0.0.1:51322_ C2532_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:51322_, base_url=http://127.0.0.1:51322}
[junit4:junit4] 2> 457925 T2407 C2532 P51322 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:52519/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 457925 T2407 C2532 P51322 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:51322 START replicas=[http://127.0.0.1:52519/collection1/] nUpdates=100
[junit4:junit4] 2> 457926 T2407 C2532 P51322 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 457926 T2407 C2532 P51322 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 457927 T2407 C2532 P51322 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 457927 T2407 C2532 P51322 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 457927 T2407 C2532 P51322 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 457927 T2407 C2532 P51322 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:52519/collection1/. core=collection1
[junit4:junit4] 2> 457928 T2407 C2532 P51322 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2533 name=collection1 org.apache.solr.core.SolrCore@28c8318b url=http://127.0.0.1:52519/collection1 node=127.0.0.1:52519_ C2533_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52519_, base_url=http://127.0.0.1:52519, leader=true}
[junit4:junit4] 2> 457939 T2350 C2533 P52519 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> ASYNC NEW_CORE C2534 name=collection1 org.apache.solr.core.SolrCore@28c8318b url=http://127.0.0.1:52519/collection1 node=127.0.0.1:52519_ C2534_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:52519_, base_url=http://127.0.0.1:52519, leader=true}
[junit4:junit4] 2> 457951 T2349 C2534 P52519 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 457955 T2349 C2534 P52519 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 457956 T2349 C2534 P52519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 457961 T2349 C2534 P52519 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 457962 T2349 C2534 P52519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 457963 T2349 C2534 P52519 oass.SolrIndexSearcher.<init> Opening Searcher@53899ce realtime
[junit4:junit4] 2> 457963 T2349 C2534 P52519 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 457963 T2349 C2534 P52519 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4] 2> ASYNC NEW_CORE C2535 name=collection1 org.apache.solr.core.SolrCore@15683790 url=http://127.0.0.1:51322/collection1 node=127.0.0.1:51322_ C2535_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:51322_, base_url=http://127.0.0.1:51322}
[junit4:junit4] 2> 457964 T2407 C2535 P51322 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 457964 T2407 C2535 P51322 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 457967 T2351 C2534 P52519 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 457967 T2351 C2534 P52519 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4] 2> 457967 T2407 C2535 P51322 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 457967 T2407 C2535 P51322 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 457968 T2407 C2535 P51322 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 457970 T2350 C2534 P52519 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 457971 T2407 C2535 P51322 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 457972 T2407 C2535 P51322 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index.20130616060241752
[junit4:junit4] 2> 457972 T2407 C2535 P51322 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index.20130616060241752 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e5599ff fullCopy=false
[junit4:junit4] 2> 457974 T2349 C2534 P52519 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 457975 T2407 C2535 P51322 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 457982 T2407 C2535 P51322 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 457982 T2407 C2535 P51322 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 457986 T2407 C2535 P51322 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 457986 T2407 C2535 P51322 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 457987 T2407 C2535 P51322 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 457987 T2407 C2535 P51322 oass.SolrIndexSearcher.<init> Opening Searcher@135543ea main
[junit4:junit4] 2> 457988 T2406 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@135543ea main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 457989 T2407 C2535 P51322 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index.20130616060241752 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index.20130616060241752;done=true>>]
[junit4:junit4] 2> 457989 T2407 C2535 P51322 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index.20130616060241752
[junit4:junit4] 2> 457990 T2407 C2535 P51322 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty4/index.20130616060241752
[junit4:junit4] 2> 457990 T2407 C2535 P51322 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 457991 T2407 C2535 P51322 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 457991 T2407 C2535 P51322 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 457991 T2407 C2535 P51322 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 457994 T2407 C2535 P51322 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 458280 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 458282 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32871_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32871"}
[junit4:junit4] 2> 458287 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:51322_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:51322"}
[junit4:junit4] 2> 458307 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458307 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458307 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458307 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458307 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458307 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458307 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458308 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458577 T2366 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 458578 T2366 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:32871_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 458717 T2314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55840_
[junit4:junit4] 2> 458719 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55840_
[junit4:junit4] 2> 458723 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 458723 T2343 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 458724 T2438 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 458724 T2372 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 458724 T2421 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 458724 T2388 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 458724 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458724 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458726 T2404 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 458725 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458725 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458725 T2336 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 458724 T2356 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 458724 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458729 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458728 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 458734 T2441 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 458735 T2441 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 459816 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 459817 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55840_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55840"}
[junit4:junit4] 2> 459817 T2337 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 459818 T2337 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 459823 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459823 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459823 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459823 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459823 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459823 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459824 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 459824 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> ASYNC NEW_CORE C2536 name=collection1 org.apache.solr.core.SolrCore@5bb83605 url=http://127.0.0.1:32871/collection1 node=127.0.0.1:32871_ C2536_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:32871_, base_url=http://127.0.0.1:32871}
[junit4:junit4] 2> 460579 T2424 C2536 P32871 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:46325/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 460580 T2424 C2536 P32871 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:32871 START replicas=[http://127.0.0.1:46325/collection1/] nUpdates=100
[junit4:junit4] 2> 460580 T2424 C2536 P32871 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 460580 T2424 C2536 P32871 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 460580 T2424 C2536 P32871 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 460580 T2424 C2536 P32871 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 460580 T2424 C2536 P32871 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 460581 T2424 C2536 P32871 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:46325/collection1/. core=collection1
[junit4:junit4] 2> 460581 T2424 C2536 P32871 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C2537 name=collection1 org.apache.solr.core.SolrCore@3175ad7c url=http://127.0.0.1:46325/collection1 node=127.0.0.1:46325_ C2537_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:46325_, base_url=http://127.0.0.1:46325, leader=true}
[junit4:junit4] 2> 460585 T2366 C2537 P46325 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 460585 T2368 C2537 P46325 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 460588 T2368 C2537 P46325 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 460588 T2368 C2537 P46325 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 460591 T2368 C2537 P46325 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 460592 T2368 C2537 P46325 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 460592 T2368 C2537 P46325 oass.SolrIndexSearcher.<init> Opening Searcher@5eb5756a realtime
[junit4:junit4] 2> 460593 T2368 C2537 P46325 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 460593 T2368 C2537 P46325 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4] 2> 460594 T2424 C2536 P32871 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 460594 T2424 C2536 P32871 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 460595 T2367 C2537 P46325 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 460596 T2367 C2537 P46325 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 460596 T2424 C2536 P32871 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 460596 T2424 C2536 P32871 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 460596 T2424 C2536 P32871 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 460598 T2366 C2537 P46325 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 460599 T2424 C2536 P32871 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 460600 T2424 C2536 P32871 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index.20130616060244380
[junit4:junit4] 2> 460600 T2424 C2536 P32871 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index.20130616060244380 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a000b16 fullCopy=false
[junit4:junit4] 2> 460602 T2368 C2537 P46325 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 460603 T2424 C2536 P32871 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 460612 T2424 C2536 P32871 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 460612 T2424 C2536 P32871 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 460614 T2424 C2536 P32871 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 460614 T2424 C2536 P32871 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 460614 T2424 C2536 P32871 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 460615 T2424 C2536 P32871 oass.SolrIndexSearcher.<init> Opening Searcher@71ef5b6d main
[junit4:junit4] 2> 460615 T2423 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@71ef5b6d main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 460616 T2424 C2536 P32871 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index.20130616060244380 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index.20130616060244380;done=true>>]
[junit4:junit4] 2> 460616 T2424 C2536 P32871 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index.20130616060244380
[junit4:junit4] 2> 460616 T2424 C2536 P32871 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty5/index.20130616060244380
[junit4:junit4] 2> 460617 T2424 C2536 P32871 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 460617 T2424 C2536 P32871 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 460617 T2424 C2536 P32871 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 460617 T2424 C2536 P32871 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 460619 T2424 C2536 P32871 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 460737 T2441 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356/collection1
[junit4:junit4] 2> 460737 T2441 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 460738 T2441 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 460738 T2441 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 460739 T2441 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356/collection1/'
[junit4:junit4] 2> 460740 T2441 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356/collection1/lib/README' to classloader
[junit4:junit4] 2> 460740 T2441 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 460779 T2441 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 460810 T2441 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 460811 T2441 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 460815 T2441 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 461129 T2441 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 461134 T2441 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 461136 T2441 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 461145 T2441 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 461148 T2441 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 461150 T2441 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 461151 T2441 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 461151 T2441 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 461151 T2441 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 461152 T2441 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 461152 T2441 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 461152 T2441 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 461152 T2441 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371337361356/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty6/
[junit4:junit4] 2> 461152 T2441 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@443c86bc
[junit4:junit4] 2> 461153 T2441 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 461153 T2441 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty6
[junit4:junit4] 2> 461153 T2441 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty6/index/
[junit4:junit4] 2> 461154 T2441 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 461154 T2441 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty6/index
[junit4:junit4] 2> 461173 T2441 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 461173 T2441 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 461176 T2441 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 461176 T2441 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 461177 T2441 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 461178 T2441 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 461179 T2441 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 461179 T2441 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 461180 T2441 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 461181 T2441 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 461181 T2441 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 461183 T2441 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 461186 T2441 oass.SolrIndexSearcher.<init> Opening Searcher@593bcd6e main
[junit4:junit4] 2> 461186 T2441 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 461186 T2441 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 461191 T2444 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@593bcd6e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 461194 T2441 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 461194 T2441 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55840 collection:collection1 shard:shard3
[junit4:junit4] 2> 461199 T2441 oasc.ZkController.register We are http://127.0.0.1:55840/collection1/ and leader is http://127.0.0.1:56083/collection1/
[junit4:junit4] 2> 461199 T2441 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55840
[junit4:junit4] 2> 461200 T2441 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 461200 T2441 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2538 name=collection1 org.apache.solr.core.SolrCore@3c639e2c url=http://127.0.0.1:55840/collection1 node=127.0.0.1:55840_ C2538_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:55840_, base_url=http://127.0.0.1:55840}
[junit4:junit4] 2> 461201 T2445 C2538 P55840 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 461201 T2441 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 461201 T2445 C2538 P55840 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 461202 T2445 C2538 P55840 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 461202 T2445 C2538 P55840 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 461203 T2314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 461204 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 461204 T2445 C2538 P55840 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 461204 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 461213 T2381 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 461277 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 461280 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53205
[junit4:junit4] 2> 461280 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 461280 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 461281 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992
[junit4:junit4] 2> 461281 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992/solr.xml
[junit4:junit4] 2> 461281 T2314 oasc.CoreContainer.<init> New CoreContainer 171036891
[junit4:junit4] 2> 461282 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992/'
[junit4:junit4] 2> 461282 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992/'
[junit4:junit4] 2> 461326 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 461327 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 461327 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 461327 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 461328 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 461328 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 461329 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 461329 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 461329 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 461329 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 461330 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 461330 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:32871_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:32871"}
[junit4:junit4] 2> 461333 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55840_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55840"}
[junit4:junit4] 2> 461333 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 461334 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 461334 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 461335 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 461336 T2457 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@546a23ef name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 461336 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 461337 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 461337 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 461337 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 461337 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 461337 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 461337 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 461337 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 461337 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 461338 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 461342 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 461343 T2459 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a44c68a name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 461344 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 461347 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 462213 T2381 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 462214 T2381 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:55840_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 462350 T2314 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53205_
[junit4:junit4] 2> 462352 T2314 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53205_
[junit4:junit4] 2> 462354 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 462355 T2336 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462355 T2459 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462355 T2356 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462355 T2343 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462355 T2438 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462355 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462355 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462355 T2372 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462355 T2421 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462355 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462355 T2388 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462357 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462357 T2404 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 462356 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462356 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462356 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462360 T2460 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 462361 T2460 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 462841 T2337 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 462842 T2337 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53205_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53205"}
[junit4:junit4] 2> 462842 T2337 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 462842 T2337 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 462845 T2438 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462845 T2459 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462846 T2372 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462846 T2343 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462846 T2336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462846 T2404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462846 T2356 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462846 T2421 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 462846 T2388 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 463363 T2460 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992/collection1
[junit4:junit4] 2> 463363 T2460 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 463364 T2460 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 463364 T2460 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 463365 T2460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992/collection1/'
[junit4:junit4] 2> 463366 T2460 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992/collection1/lib/README' to classloader
[junit4:junit4] 2> 463366 T2460 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 463394 T2460 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 463420 T2460 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 463421 T2460 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 463424 T2460 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 463726 T2460 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 463731 T2460 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 463733 T2460 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 463742 T2460 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 463744 T2460 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 463747 T2460 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 463748 T2460 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 463748 T2460 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 463748 T2460 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 463749 T2460 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 463749 T2460 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 463749 T2460 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 463750 T2460 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371337364992/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty7/
[junit4:junit4] 2> 463750 T2460 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@443c86bc
[junit4:junit4] 2> 463750 T2460 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 463751 T2460 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty7
[junit4:junit4] 2> 463751 T2460 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty7/index/
[junit4:junit4] 2> 463751 T2460 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 463751 T2460 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty7/index
[junit4:junit4] 2> 463754 T2460 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371337343382/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 463754 T2460 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 463756 T2460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 463756 T2460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 463757 T2460 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 463757 T2460 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 463757 T2460 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 463758 T2460 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 463758 T2460 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 463758 T2460 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 463758 T2460 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 463761 T2460 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 463763 T2460 oass.SolrIndexSearcher.<init> Opening Searcher@72c35eec main
[junit4:junit4] 2> 463764 T2460 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 463764 T2460 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 463775 T2461 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@72c35eec main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 463776 T2460 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 463776 T2460 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53205 collection:collection1 shard:shard1
[junit4:junit4] 2> 463779 T2460 oasc.ZkController.register We are http://127.0.0.1:53205/collection1/ and leader is http://127.0.0.1:52519/collection1/
[junit4:junit4] 2> 463779 T2460 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53205
[junit4:junit4] 2> 463779 T2460 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 463779 T2460 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C2539 name=collection1 org.apache.solr.core.SolrCore@59546b8d url=http://127.0.0.1:53205/collection1 node=127.0.0.1:53205_ C2539_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:53205_, base_url=http://127.0.0.1:53205}
[junit4:junit4] 2> 463780 T2462 C2539 P53205 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 463780 T2460 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 463780 T2462 C2539 P53205 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 463781 T2462 C2539 P53205 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 463781 T2462 C2539 P53205 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 463781 T2314 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 463782 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 463782 T2462 C2539 P53205 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 463782 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 463786 T2351 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 8, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 463846 T2314 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 463848 T2314 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36143
[junit4:junit4] 2> 463849 T2314 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 463849 T2314 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 463850 T2314 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1371337367566
[junit4:junit4] 2> 463850 T2314 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1371337367566/solr.xml
[junit4:junit4] 2> 463850 T2314 oasc.CoreContainer.<init> New CoreContainer 1841499314
[junit4:junit4] 2> 463851 T2314 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1371337367566/'
[junit4:junit4] 2> 463851 T2314 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1371337367566/'
[junit4:junit4] 2> 463891 T2314 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 463891 T2314 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 463892 T2314 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 463892 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 463892 T2314 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 463893 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 463893 T2314 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 463893 T2314 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 463894 T2314 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 463894 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 463897 T2314 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 463898 T2314 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39914/solr
[junit4:junit4] 2> 463898 T2314 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 463899 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 463900 T2474 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61844710 name:ZooKeeperConnection Watcher:127.0.0.1:39914 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 463901 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 463902 T2314 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 463905 T2314 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 463906 T2476 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41818bb1 name:ZooKeeperConnection Watcher:127.0.0.1:39914/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 463907 T2314 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 463909 T2314 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C2540 name=collection1 org.apache.solr.core.SolrCore@3c639e2c url=http://127.0.0.1:55840/collection1 node=127.0.0.1:55840_ C2540_STATE=coll:collectio
[...truncated too long message...]
tor.java:956)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 71) Thread[id=2410, name=qtp1760481929-2410 Selector0, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method)
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:274)
[junit4:junit4] 2> at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193)
[junit4:junit4] 2> at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210)
[junit4:junit4] 2> at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216)
[junit4:junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:80)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 72) Thread[id=2611, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[93165A8081EFE724]-SendThread(localhost.localdomain:39914), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
[junit4:junit4] 2> 73) Thread[id=2395, name=qtp1941881954-2395 Acceptor0 SelectChannelConnector@127.0.0.1:51322, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at org.eclipse.jetty.server.nio.SelectChannelConnector.getConnection(SelectChannelConnector.java:160)
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:934)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 74) Thread[id=2520, name=Thread-1124, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Object.wait(Native Method)
[junit4:junit4] 2> at java.lang.Object.wait(Object.java:503)
[junit4:junit4] 2> at org.apache.solr.core.CloserThread.run(CoreContainer.java:1260)
[junit4:junit4] 2> 75) Thread[id=2423, name=searcherExecutor-1077-thread-1, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4:junit4] 2> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
[junit4:junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
[junit4:junit4] 2> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 76) Thread[id=2532, name=TEST-ChaosMonkeySafeLeaderTest.testDistribSearch-seed#[93165A8081EFE724]-SendThread(localhost.localdomain:39914), state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at java.lang.Thread.sleep(Native Method)
[junit4:junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
[junit4:junit4] 2> 77) Thread[id=2451, name=qtp1131585113-2451 Acceptor1 SelectChannelConnector@127.0.0.1:53205, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:956)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 78) Thread[id=2378, name=qtp1564835914-2378 Selector1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
[junit4:junit4] 2> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
[junit4:junit4] 2> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
[junit4:junit4] 2> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:106)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:536)
[junit4:junit4] 2> at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=zh_SG, timezone=Etc/GMT-7
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=98,free=24533552,total=196014080
[junit4:junit4] 2> NOTE: All tests run in this JVM: [QueryResultKeyTest, TermVectorComponentTest, CoreAdminHandlerTest, BinaryUpdateRequestHandlerTest, SOLR749Test, TestSolrDeletionPolicy1, SolrIndexSplitterTest, TestRemoteStreaming, TestConfig, OverseerTest, TestCloudManagedSchema, AutoCommitTest, TestSolrCoreProperties, SynonymTokenizerTest, WordBreakSolrSpellCheckerTest, SoftAutoCommitTest, OverseerCollectionProcessorTest, TestIBSimilarityFactory, TestPerFieldSimilarity, TestCharFilters, StatelessScriptUpdateProcessorFactoryTest, MBeansHandlerTest, TestFuzzyAnalyzedSuggestions, PluginInfoTest, ShowFileRequestHandlerTest, TestStressVersions, TestLFUCache, PreAnalyzedFieldTest, TestGroupingSearch, ResponseLogComponentTest, TestSolrIndexConfig, SliceStateUpdateTest, SliceStateTest, TestStressRecovery, CircularListTest, TestFieldCollectionResource, TestRealTimeGet, TestQueryUtils, SyncSliceTest, TestFastWriter, SpatialFilterTest, TestFunctionQuery, UUIDFieldTest, PrimUtilsTest, SearchHandlerTest, DirectSolrConnectionTest, ClusterStateUpdateTest, TestReplicationHandler, TestMaxScoreQueryParser, TestSolrDeletionPolicy2, SimpleFacetsTest, RegexBoostProcessorTest, TestPropInjectDefaults, TestPHPSerializedResponseWriter, LeaderElectionTest, JsonLoaderTest, LoggingHandlerTest, PeerSyncTest, TestDocSet, TestValueSourceCache, QueryParsingTest, TestSchemaResource, ResourceLoaderTest, TestXIncludeConfig, TestSolrQueryParser, TestSurroundQueryParser, FastVectorHighlighterTest, SystemInfoHandlerTest, MinimalSchemaTest, FieldMutatingUpdateProcessorTest, TestCoreContainer, AlternateDirectoryTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=93165A8081EFE724 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=zh_SG -Dtests.timezone=Etc/GMT-7 -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR 0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4:junit4] > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([93165A8081EFE724]:0)
[junit4:junit4] Completed on J0 in 7225.78s, 1 test, 2 errors <<< FAILURES!
[...truncated 11 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 297 suites, 1246 tests, 1 suite-level error, 1 error, 13 ignored (7 assumptions)
Total time: 144 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 6086
- Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6086/
Java: 32bit/jdk1.6.0_45 -client -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 227 from http://127.0.0.1:33622/collection1lastClient and got 237 from http://127.0.0.1:57036/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 227 from http://127.0.0.1:33622/collection1lastClient and got 237 from http://127.0.0.1:57036/collection1
at __randomizedtesting.SeedInfo.seed([47A3AE6B36F45749:C645207341AB3775]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:662)
Build Log:
[...truncated 9030 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 572827 T2041 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4] 2> 572833 T2041 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1371349508364
[junit4:junit4] 2> 572834 T2041 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 572835 T2042 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 572935 T2041 oasc.ZkTestServer.run start zk server on port:54854
[junit4:junit4] 2> 572936 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 573088 T2048 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@fef62b name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 573088 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 573089 T2041 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 573093 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 573095 T2050 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@142b76d name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 573096 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 573096 T2041 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 573098 T2041 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 573101 T2041 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 573102 T2041 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 573105 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 573105 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 573108 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 573109 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 573111 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 573112 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 573114 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 573114 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 573116 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 573117 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 573119 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 573119 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 573121 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 573122 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 573125 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 573125 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 573127 T2041 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 573128 T2041 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 573221 T2041 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 573275 T2041 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41687
[junit4:junit4] 2> 573277 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 573277 T2041 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 573278 T2041 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661
[junit4:junit4] 2> 573278 T2041 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661/solr.xml
[junit4:junit4] 2> 573279 T2041 oasc.CoreContainer.<init> New CoreContainer 6041077
[junit4:junit4] 2> 573280 T2041 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661/'
[junit4:junit4] 2> 573280 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661/'
[junit4:junit4] 2> 573364 T2041 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 573365 T2041 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 573366 T2041 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 573366 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 573367 T2041 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 573368 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 573368 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 573369 T2041 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 573369 T2041 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 573370 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 573379 T2041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 573380 T2041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54854/solr
[junit4:junit4] 2> 573381 T2041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 573382 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 573383 T2061 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12e54bf name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 573383 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 573385 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 573388 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 573389 T2063 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9ff3ce name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 573389 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 573390 T2041 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 573393 T2041 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 573395 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 573397 T2041 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41687_
[junit4:junit4] 2> 573398 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41687_
[junit4:junit4] 2> 573401 T2041 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 573412 T2041 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 573414 T2041 oasc.Overseer.start Overseer (id=89872761380470787-127.0.0.1:41687_-n_0000000000) starting
[junit4:junit4] 2> 573416 T2041 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 573419 T2065 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 573420 T2041 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 573422 T2041 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 573423 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 573426 T2064 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 573430 T2066 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 573431 T2066 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 574930 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 574931 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41687_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41687"}
[junit4:junit4] 2> 574931 T2064 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 574931 T2064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 574935 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 575432 T2066 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661/collection1
[junit4:junit4] 2> 575432 T2066 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 575433 T2066 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 575433 T2066 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 575434 T2066 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661/collection1/'
[junit4:junit4] 2> 575435 T2066 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661/collection1/lib/README' to classloader
[junit4:junit4] 2> 575436 T2066 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 575494 T2066 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 575579 T2066 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 575580 T2066 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 575588 T2066 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 576436 T2066 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 576444 T2066 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 576449 T2066 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 576462 T2066 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 576467 T2066 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 576471 T2066 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 576472 T2066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 576472 T2066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 576473 T2066 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 576474 T2066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 576474 T2066 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 576475 T2066 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 576488 T2066 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371349508661/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/control/data/
[junit4:junit4] 2> 576495 T2066 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11ef443
[junit4:junit4] 2> 576495 T2066 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 576496 T2066 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/control/data
[junit4:junit4] 2> 576497 T2066 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/control/data/index/
[junit4:junit4] 2> 576497 T2066 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 576498 T2066 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/control/data/index
[junit4:junit4] 2> 576504 T2066 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 576505 T2066 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 576508 T2066 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 576508 T2066 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 576509 T2066 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 576509 T2066 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 576510 T2066 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 576510 T2066 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 576511 T2066 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 576512 T2066 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 576512 T2066 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 576514 T2066 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 576518 T2066 oass.SolrIndexSearcher.<init> Opening Searcher@103bb25 main
[junit4:junit4] 2> 576519 T2066 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 576520 T2066 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 576526 T2067 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@103bb25 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 576528 T2066 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 576529 T2066 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41687 collection:control_collection shard:shard1
[junit4:junit4] 2> 576529 T2066 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 576535 T2066 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 576538 T2066 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 576539 T2066 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 576539 T2066 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41687/collection1/
[junit4:junit4] 2> 576540 T2066 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 576540 T2066 oasc.SyncStrategy.syncToMe http://127.0.0.1:41687/collection1/ has no replicas
[junit4:junit4] 2> 576540 T2066 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41687/collection1/
[junit4:junit4] 2> 576541 T2066 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 577940 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 577949 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 577998 T2066 oasc.ZkController.register We are http://127.0.0.1:41687/collection1/ and leader is http://127.0.0.1:41687/collection1/
[junit4:junit4] 2> 577999 T2066 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41687
[junit4:junit4] 2> 577999 T2066 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 577999 T2066 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 577999 T2066 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 578001 T2066 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 578002 T2041 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 578002 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 578003 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 578006 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 578007 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 578008 T2070 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16eb49c name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 578008 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 578009 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 578011 T2041 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 578105 T2041 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 578108 T2041 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50281
[junit4:junit4] 2> 578109 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 578110 T2041 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 578110 T2041 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542
[junit4:junit4] 2> 578111 T2041 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542/solr.xml
[junit4:junit4] 2> 578111 T2041 oasc.CoreContainer.<init> New CoreContainer 8501002
[junit4:junit4] 2> 578112 T2041 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542/'
[junit4:junit4] 2> 578113 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542/'
[junit4:junit4] 2> 578190 T2041 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 578191 T2041 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 578191 T2041 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 578192 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 578192 T2041 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 578193 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 578193 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 578193 T2041 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 578194 T2041 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 578194 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 578198 T2041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 578199 T2041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54854/solr
[junit4:junit4] 2> 578200 T2041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 578201 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 578202 T2081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e5979b name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 578203 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 578205 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 578208 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 578219 T2083 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39ea06 name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 578220 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 578224 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 579227 T2041 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50281_
[junit4:junit4] 2> 579229 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50281_
[junit4:junit4] 2> 579231 T2063 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 579231 T2070 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 579231 T2083 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 579231 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 579245 T2084 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 579245 T2084 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 579453 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 579453 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:41687_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:41687"}
[junit4:junit4] 2> 579455 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50281_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50281"}
[junit4:junit4] 2> 579456 T2064 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 579456 T2064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 579460 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 579460 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 579460 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 580246 T2084 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542/collection1
[junit4:junit4] 2> 580247 T2084 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 580248 T2084 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 580248 T2084 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 580249 T2084 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542/collection1/'
[junit4:junit4] 2> 580250 T2084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542/collection1/lib/README' to classloader
[junit4:junit4] 2> 580250 T2084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 580298 T2084 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 580352 T2084 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 580353 T2084 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 580358 T2084 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 580949 T2084 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 580956 T2084 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 580959 T2084 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 580975 T2084 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 580980 T2084 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 580984 T2084 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 580985 T2084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 580986 T2084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 580987 T2084 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 580990 T2084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 580990 T2084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 580990 T2084 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 580991 T2084 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371349513542/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1/
[junit4:junit4] 2> 580991 T2084 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11ef443
[junit4:junit4] 2> 580992 T2084 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 580993 T2084 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1
[junit4:junit4] 2> 580993 T2084 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1/index/
[junit4:junit4] 2> 580994 T2084 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 580994 T2084 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1/index
[junit4:junit4] 2> 580999 T2084 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 581000 T2084 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 581002 T2084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 581003 T2084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 581004 T2084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 581005 T2084 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 581006 T2084 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 581006 T2084 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 581007 T2084 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 581008 T2084 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 581008 T2084 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 581012 T2084 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 581017 T2084 oass.SolrIndexSearcher.<init> Opening Searcher@1e961ab main
[junit4:junit4] 2> 581018 T2084 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 581018 T2084 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 581034 T2085 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e961ab main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 581036 T2084 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 581037 T2084 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50281 collection:collection1 shard:shard1
[junit4:junit4] 2> 581037 T2084 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 581042 T2084 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 581044 T2084 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 581044 T2084 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 581044 T2084 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50281/collection1/
[junit4:junit4] 2> 581044 T2084 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 581045 T2084 oasc.SyncStrategy.syncToMe http://127.0.0.1:50281/collection1/ has no replicas
[junit4:junit4] 2> 581045 T2084 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50281/collection1/
[junit4:junit4] 2> 581045 T2084 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 582467 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 582473 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 582473 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 582473 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 582654 T2084 oasc.ZkController.register We are http://127.0.0.1:50281/collection1/ and leader is http://127.0.0.1:50281/collection1/
[junit4:junit4] 2> 582655 T2084 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50281
[junit4:junit4] 2> 582655 T2084 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 582655 T2084 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 582656 T2084 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 582657 T2084 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 582658 T2041 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 582658 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 582659 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 582743 T2041 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 582745 T2041 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57342
[junit4:junit4] 2> 582746 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 582746 T2041 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 582746 T2041 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192
[junit4:junit4] 2> 582747 T2041 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192/solr.xml
[junit4:junit4] 2> 582747 T2041 oasc.CoreContainer.<init> New CoreContainer 4788968
[junit4:junit4] 2> 582748 T2041 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192/'
[junit4:junit4] 2> 582748 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192/'
[junit4:junit4] 2> 582828 T2041 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 582829 T2041 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 582830 T2041 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 582830 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 582831 T2041 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 582831 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 582832 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 582832 T2041 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 582832 T2041 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 582833 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 582836 T2041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 582837 T2041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54854/solr
[junit4:junit4] 2> 582837 T2041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 582838 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 582840 T2097 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@db2f36 name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 582840 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 582841 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 582844 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 582845 T2099 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17c9c58 name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 582845 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 582848 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 583850 T2041 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57342_
[junit4:junit4] 2> 583852 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57342_
[junit4:junit4] 2> 583854 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 583854 T2083 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 583854 T2063 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 583855 T2099 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 583855 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 583855 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 583856 T2070 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 583860 T2100 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 583861 T2100 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 584141 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 584142 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:50281_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:50281"}
[junit4:junit4] 2> 584146 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57342_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57342"}
[junit4:junit4] 2> 584146 T2064 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 584146 T2064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 584149 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 584149 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 584149 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 584149 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 584862 T2100 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192/collection1
[junit4:junit4] 2> 584862 T2100 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 584863 T2100 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 584863 T2100 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 584864 T2100 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192/collection1/'
[junit4:junit4] 2> 584865 T2100 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192/collection1/lib/README' to classloader
[junit4:junit4] 2> 584865 T2100 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 584905 T2100 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 584973 T2100 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 584974 T2100 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 584979 T2100 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 585493 T2100 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 585504 T2100 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 585507 T2100 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 585516 T2100 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 585519 T2100 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 585522 T2100 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 585522 T2100 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 585523 T2100 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 585523 T2100 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 585524 T2100 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 585524 T2100 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 585524 T2100 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 585524 T2100 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371349518192/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2/
[junit4:junit4] 2> 585525 T2100 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11ef443
[junit4:junit4] 2> 585525 T2100 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 585525 T2100 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2
[junit4:junit4] 2> 585526 T2100 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2/index/
[junit4:junit4] 2> 585526 T2100 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 585526 T2100 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2/index
[junit4:junit4] 2> 585530 T2100 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 585530 T2100 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 585532 T2100 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 585532 T2100 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 585532 T2100 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 585533 T2100 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 585534 T2100 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 585534 T2100 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 585534 T2100 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 585535 T2100 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 585535 T2100 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 585537 T2100 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 585541 T2100 oass.SolrIndexSearcher.<init> Opening Searcher@28973c main
[junit4:junit4] 2> 585541 T2100 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 585541 T2100 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 585548 T2101 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@28973c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 585551 T2100 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 585551 T2100 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57342 collection:collection1 shard:shard2
[junit4:junit4] 2> 585552 T2100 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 585557 T2100 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 585558 T2100 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 585559 T2100 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 585559 T2100 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:57342/collection1/
[junit4:junit4] 2> 585559 T2100 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 585560 T2100 oasc.SyncStrategy.syncToMe http://127.0.0.1:57342/collection1/ has no replicas
[junit4:junit4] 2> 585560 T2100 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:57342/collection1/
[junit4:junit4] 2> 585560 T2100 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 585655 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 585660 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 585660 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 585660 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 585660 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 585673 T2100 oasc.ZkController.register We are http://127.0.0.1:57342/collection1/ and leader is http://127.0.0.1:57342/collection1/
[junit4:junit4] 2> 585673 T2100 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57342
[junit4:junit4] 2> 585673 T2100 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 585674 T2100 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 585674 T2100 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 585675 T2100 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 585675 T2041 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 585676 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 585676 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 585752 T2041 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 585754 T2041 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49705
[junit4:junit4] 2> 585755 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 585755 T2041 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 585756 T2041 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209
[junit4:junit4] 2> 585756 T2041 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209/solr.xml
[junit4:junit4] 2> 585756 T2041 oasc.CoreContainer.<init> New CoreContainer 25722093
[junit4:junit4] 2> 585757 T2041 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209/'
[junit4:junit4] 2> 585757 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209/'
[junit4:junit4] 2> 585824 T2041 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 585824 T2041 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 585825 T2041 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 585825 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 585825 T2041 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 585826 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 585826 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 585826 T2041 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 585827 T2041 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 585827 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 585831 T2041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 585831 T2041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54854/solr
[junit4:junit4] 2> 585832 T2041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 585833 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 585834 T2113 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@209a6e name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 585835 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 585836 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 585840 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 585840 T2115 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c56c9 name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 585841 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 585845 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 586847 T2041 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49705_
[junit4:junit4] 2> 586848 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49705_
[junit4:junit4] 2> 586850 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 586850 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 586850 T2063 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 586851 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 586851 T2115 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 586850 T2083 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 586852 T2070 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 586851 T2099 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 586852 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 586856 T2116 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 586857 T2116 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 587164 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 587165 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:57342_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:57342"}
[junit4:junit4] 2> 587167 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49705_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49705"}
[junit4:junit4] 2> 587168 T2064 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 587168 T2064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 587171 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 587171 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 587171 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 587171 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 587171 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 587858 T2116 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209/collection1
[junit4:junit4] 2> 587858 T2116 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 587859 T2116 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 587859 T2116 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 587860 T2116 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209/collection1/'
[junit4:junit4] 2> 587860 T2116 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209/collection1/lib/README' to classloader
[junit4:junit4] 2> 587861 T2116 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 587901 T2116 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 587955 T2116 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 587957 T2116 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 587963 T2116 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 588480 T2116 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 588487 T2116 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 588499 T2116 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 588508 T2116 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 588511 T2116 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 588514 T2116 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 588515 T2116 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 588515 T2116 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 588515 T2116 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 588516 T2116 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 588516 T2116 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 588517 T2116 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 588517 T2116 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371349521209/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3/
[junit4:junit4] 2> 588517 T2116 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11ef443
[junit4:junit4] 2> 588518 T2116 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 588518 T2116 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3
[junit4:junit4] 2> 588518 T2116 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3/index/
[junit4:junit4] 2> 588519 T2116 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 588519 T2116 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3/index
[junit4:junit4] 2> 588522 T2116 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 588522 T2116 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 588524 T2116 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 588524 T2116 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 588524 T2116 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 588525 T2116 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 588526 T2116 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 588526 T2116 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 588526 T2116 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 588527 T2116 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 588527 T2116 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 588529 T2116 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 588532 T2116 oass.SolrIndexSearcher.<init> Opening Searcher@e38729 main
[junit4:junit4] 2> 588533 T2116 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 588533 T2116 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 588538 T2117 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e38729 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 588540 T2116 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 588540 T2116 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49705 collection:collection1 shard:shard3
[junit4:junit4] 2> 588541 T2116 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 588548 T2116 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 588583 T2116 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 588584 T2116 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 588584 T2116 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49705/collection1/
[junit4:junit4] 2> 588584 T2116 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 588584 T2116 oasc.SyncStrategy.syncToMe http://127.0.0.1:49705/collection1/ has no replicas
[junit4:junit4] 2> 588585 T2116 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49705/collection1/
[junit4:junit4] 2> 588585 T2116 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 588675 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 588680 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 588680 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 588680 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 588680 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 588680 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 588688 T2116 oasc.ZkController.register We are http://127.0.0.1:49705/collection1/ and leader is http://127.0.0.1:49705/collection1/
[junit4:junit4] 2> 588688 T2116 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49705
[junit4:junit4] 2> 588688 T2116 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 588688 T2116 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 588689 T2116 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 588690 T2116 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 588690 T2041 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 588691 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 588691 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 588775 T2041 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 588777 T2041 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37161
[junit4:junit4] 2> 588777 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 588778 T2041 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 588778 T2041 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225
[junit4:junit4] 2> 588778 T2041 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225/solr.xml
[junit4:junit4] 2> 588779 T2041 oasc.CoreContainer.<init> New CoreContainer 19096697
[junit4:junit4] 2> 588779 T2041 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225/'
[junit4:junit4] 2> 588780 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225/'
[junit4:junit4] 2> 588842 T2041 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 588843 T2041 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 588843 T2041 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 588844 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 588844 T2041 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 588845 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 588845 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 588845 T2041 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 588846 T2041 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 588846 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 588849 T2041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 588850 T2041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54854/solr
[junit4:junit4] 2> 588850 T2041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 588851 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 588852 T2129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f49bbd name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 588853 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 588854 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 588859 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 588860 T2131 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8512a name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 588860 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 588862 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 589865 T2041 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37161_
[junit4:junit4] 2> 589867 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37161_
[junit4:junit4] 2> 589869 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 589869 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 589869 T2063 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 589869 T2083 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 589869 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 589870 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 589870 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 589871 T2099 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 589870 T2131 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 589871 T2115 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 589872 T2070 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 589876 T2132 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 589877 T2132 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 590185 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 590186 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49705_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49705"}
[junit4:junit4] 2> 590189 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37161_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37161"}
[junit4:junit4] 2> 590190 T2064 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 590190 T2064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 590192 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 590193 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 590193 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 590193 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 590192 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 590193 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 590878 T2132 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225/collection1
[junit4:junit4] 2> 590879 T2132 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 590879 T2132 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 590880 T2132 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 590881 T2132 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225/collection1/'
[junit4:junit4] 2> 590882 T2132 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225/collection1/lib/README' to classloader
[junit4:junit4] 2> 590882 T2132 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 590937 T2132 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 591034 T2132 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 591036 T2132 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 591044 T2132 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 591716 T2132 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 591722 T2132 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 591725 T2132 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 591734 T2132 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 591742 T2132 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 591746 T2132 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 591747 T2132 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 591747 T2132 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 591748 T2132 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 591749 T2132 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 591749 T2132 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 591749 T2132 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 591750 T2132 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371349524225/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/
[junit4:junit4] 2> 591750 T2132 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11ef443
[junit4:junit4] 2> 591751 T2132 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 591751 T2132 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4
[junit4:junit4] 2> 591751 T2132 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index/
[junit4:junit4] 2> 591752 T2132 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 591752 T2132 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index
[junit4:junit4] 2> 591755 T2132 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 591755 T2132 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 591757 T2132 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 591758 T2132 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 591758 T2132 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 591759 T2132 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 591759 T2132 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 591760 T2132 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 591760 T2132 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 591761 T2132 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 591761 T2132 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 591763 T2132 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 591766 T2132 oass.SolrIndexSearcher.<init> Opening Searcher@c1a1fc main
[junit4:junit4] 2> 591767 T2132 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 591767 T2132 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 591773 T2133 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c1a1fc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 591775 T2132 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 591775 T2132 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37161 collection:collection1 shard:shard1
[junit4:junit4] 2> 591777 T2132 oasc.ZkController.register We are http://127.0.0.1:37161/collection1/ and leader is http://127.0.0.1:50281/collection1/
[junit4:junit4] 2> 591777 T2132 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37161
[junit4:junit4] 2> 591777 T2132 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 591777 T2132 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3873 name=collection1 org.apache.solr.core.SolrCore@dbc56c url=http://127.0.0.1:37161/collection1 node=127.0.0.1:37161_ C3873_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:37161_, base_url=http://127.0.0.1:37161}
[junit4:junit4] 2> 591778 T2134 C3873 P37161 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 591778 T2134 C3873 P37161 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 591778 T2132 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 591779 T2134 C3873 P37161 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 591779 T2134 C3873 P37161 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 591779 T2041 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 591780 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 591780 T2134 C3873 P37161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 591780 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 591784 T2078 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 591847 T2041 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 591859 T2041 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59158
[junit4:junit4] 2> 591860 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 591860 T2041 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 591861 T2041 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313
[junit4:junit4] 2> 591861 T2041 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313/solr.xml
[junit4:junit4] 2> 591861 T2041 oasc.CoreContainer.<init> New CoreContainer 13040081
[junit4:junit4] 2> 591862 T2041 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313/'
[junit4:junit4] 2> 591862 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313/'
[junit4:junit4] 2> 591931 T2041 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 591932 T2041 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 591932 T2041 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 591932 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 591933 T2041 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 591933 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 591934 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 591934 T2041 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 591934 T2041 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 591935 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 591938 T2041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 591939 T2041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54854/solr
[junit4:junit4] 2> 591940 T2041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 591943 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 591945 T2146 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6da83d name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 591945 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 591946 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 591949 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 591950 T2148 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f48ee2 name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 591950 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 591958 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 592960 T2041 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59158_
[junit4:junit4] 2> 592962 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59158_
[junit4:junit4] 2> 592964 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 592964 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 592964 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 592965 T2148 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 592965 T2063 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 592964 T2131 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 592966 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 592964 T2083 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 592965 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 592966 T2099 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 592966 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 592967 T2115 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 592967 T2070 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 592973 T2149 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 592973 T2149 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 593200 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 593201 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37161_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37161"}
[junit4:junit4] 2> 593204 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59158_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59158"}
[junit4:junit4] 2> 593204 T2064 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 593204 T2064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 593207 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 593207 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 593207 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 593207 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 593207 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 593207 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 593207 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 593785 T2078 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 593785 T2078 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:37161_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001
[junit4:junit4] 2> 593974 T2149 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313/collection1
[junit4:junit4] 2> 593975 T2149 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 593975 T2149 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 593975 T2149 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 593976 T2149 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313/collection1/'
[junit4:junit4] 2> 593977 T2149 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313/collection1/lib/README' to classloader
[junit4:junit4] 2> 593977 T2149 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 594018 T2149 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 594068 T2149 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 594069 T2149 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 594074 T2149 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 594632 T2149 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 594639 T2149 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 594642 T2149 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 594650 T2149 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 594653 T2149 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 594656 T2149 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 594657 T2149 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 594658 T2149 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 594658 T2149 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 594658 T2149 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 594659 T2149 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 594659 T2149 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 594659 T2149 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371349527313/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/
[junit4:junit4] 2> 594659 T2149 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11ef443
[junit4:junit4] 2> 594660 T2149 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 594660 T2149 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5
[junit4:junit4] 2> 594661 T2149 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index/
[junit4:junit4] 2> 594661 T2149 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 594661 T2149 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index
[junit4:junit4] 2> 594664 T2149 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 594664 T2149 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 594666 T2149 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 594666 T2149 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 594666 T2149 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 594667 T2149 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 594668 T2149 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 594668 T2149 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 594668 T2149 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 594674 T2149 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 594674 T2149 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 594676 T2149 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 594680 T2149 oass.SolrIndexSearcher.<init> Opening Searcher@595a8 main
[junit4:junit4] 2> 594680 T2149 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 594681 T2149 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 594685 T2150 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@595a8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 594687 T2149 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 594687 T2149 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59158 collection:collection1 shard:shard2
[junit4:junit4] 2> 594689 T2149 oasc.ZkController.register We are http://127.0.0.1:59158/collection1/ and leader is http://127.0.0.1:57342/collection1/
[junit4:junit4] 2> 594690 T2149 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59158
[junit4:junit4] 2> 594690 T2149 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 594690 T2149 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3874 name=collection1 org.apache.solr.core.SolrCore@1b672fa url=http://127.0.0.1:59158/collection1 node=127.0.0.1:59158_ C3874_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:59158_, base_url=http://127.0.0.1:59158}
[junit4:junit4] 2> 594690 T2151 C3874 P59158 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 594691 T2149 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 594691 T2151 C3874 P59158 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 594691 T2151 C3874 P59158 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 594692 T2151 C3874 P59158 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 594692 T2041 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 594693 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 594692 T2151 C3874 P59158 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 594693 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 594697 T2092 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 594711 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 594712 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59158_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59158"}
[junit4:junit4] 2> 594716 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 594716 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 594716 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 594716 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 594716 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 594716 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 594716 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 594790 T2041 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 594797 T2041 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49290
[junit4:junit4] 2> 594798 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 594798 T2041 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 594799 T2041 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227
[junit4:junit4] 2> 594799 T2041 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227/solr.xml
[junit4:junit4] 2> 594800 T2041 oasc.CoreContainer.<init> New CoreContainer 1235853
[junit4:junit4] 2> 594800 T2041 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227/'
[junit4:junit4] 2> 594802 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227/'
[junit4:junit4] 2> 594885 T2041 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 594886 T2041 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 594886 T2041 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 594887 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 594888 T2041 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 594888 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 594889 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 594889 T2041 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 594890 T2041 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 594890 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 594895 T2041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 594896 T2041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54854/solr
[junit4:junit4] 2> 594896 T2041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 594900 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 594901 T2163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@90dc2 name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 594903 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 594909 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 594915 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 594917 T2165 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da3509 name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 594917 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 594923 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 595697 T2092 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 595698 T2092 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:59158_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> ASYNC NEW_CORE C3875 name=collection1 org.apache.solr.core.SolrCore@dbc56c url=http://127.0.0.1:37161/collection1 node=127.0.0.1:37161_ C3875_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:37161_, base_url=http://127.0.0.1:37161}
[junit4:junit4] 2> 595786 T2134 C3875 P37161 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:50281/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 595787 T2134 C3875 P37161 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37161 START replicas=[http://127.0.0.1:50281/collection1/] nUpdates=100
[junit4:junit4] 2> 595787 T2134 C3875 P37161 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 595787 T2134 C3875 P37161 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 595788 T2134 C3875 P37161 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 595788 T2134 C3875 P37161 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 595788 T2134 C3875 P37161 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 595789 T2134 C3875 P37161 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:50281/collection1/. core=collection1
[junit4:junit4] 2> 595789 T2134 C3875 P37161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3876 name=collection1 org.apache.solr.core.SolrCore@17d9718 url=http://127.0.0.1:50281/collection1 node=127.0.0.1:50281_ C3876_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:50281_, base_url=http://127.0.0.1:50281, leader=true}
[junit4:junit4] 2> 595814 T2077 C3876 P50281 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 595815 T2076 C3876 P50281 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 595817 T2076 C3876 P50281 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 595818 T2076 C3876 P50281 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 595827 T2076 C3876 P50281 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 595828 T2076 C3876 P50281 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 595828 T2076 C3876 P50281 oass.SolrIndexSearcher.<init> Opening Searcher@374234 realtime
[junit4:junit4] 2> 595829 T2076 C3876 P50281 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 595829 T2076 C3876 P50281 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4] 2> ASYNC NEW_CORE C3877 name=collection1 org.apache.solr.core.SolrCore@dbc56c url=http://127.0.0.1:37161/collection1 node=127.0.0.1:37161_ C3877_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:37161_, base_url=http://127.0.0.1:37161}
[junit4:junit4] 2> 595830 T2134 C3877 P37161 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 595830 T2134 C3877 P37161 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 595832 T2079 C3876 P50281 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 595833 T2079 C3876 P50281 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 595834 T2134 C3877 P37161 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 595834 T2134 C3877 P37161 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 595834 T2134 C3877 P37161 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 595836 T2077 C3876 P50281 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 595837 T2134 C3877 P37161 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 595837 T2134 C3877 P37161 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index.20130616052531367
[junit4:junit4] 2> 595838 T2134 C3877 P37161 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index.20130616052531367 lockFactory=org.apache.lucene.store.NativeFSLockFactory@178b0de fullCopy=false
[junit4:junit4] 2> 595845 T2076 C3876 P50281 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 595846 T2134 C3877 P37161 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 595858 T2134 C3877 P37161 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 595858 T2134 C3877 P37161 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 595860 T2134 C3877 P37161 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 595861 T2134 C3877 P37161 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 595862 T2134 C3877 P37161 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 595862 T2134 C3877 P37161 oass.SolrIndexSearcher.<init> Opening Searcher@8925d7 main
[junit4:junit4] 2> 595864 T2133 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@8925d7 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 595864 T2134 C3877 P37161 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index.20130616052531367 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index.20130616052531367;done=true>>]
[junit4:junit4] 2> 595865 T2134 C3877 P37161 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index.20130616052531367
[junit4:junit4] 2> 595865 T2134 C3877 P37161 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty4/index.20130616052531367
[junit4:junit4] 2> 595866 T2134 C3877 P37161 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 595867 T2134 C3877 P37161 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 595867 T2134 C3877 P37161 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 595867 T2134 C3877 P37161 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 595887 T2134 C3877 P37161 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 595926 T2041 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49290_
[junit4:junit4] 2> 595927 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49290_
[junit4:junit4] 2> 595928 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 595928 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 595928 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 595929 T2131 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 595930 T2063 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 595930 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 595930 T2148 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 595930 T2083 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 595931 T2099 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 595931 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 595931 T2115 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 595931 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 595930 T2165 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 595930 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 595931 T2070 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 595937 T2168 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 595937 T2168 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 596219 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 596220 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37161_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37161"}
[junit4:junit4] 2> 596222 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49290_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49290"}
[junit4:junit4] 2> 596223 T2064 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 596223 T2064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 596226 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 596226 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 596226 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 596226 T2165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 596226 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 596226 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 596226 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 596226 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 596938 T2168 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227/collection1
[junit4:junit4] 2> 596939 T2168 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 596940 T2168 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 596940 T2168 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 596941 T2168 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227/collection1/'
[junit4:junit4] 2> 596942 T2168 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227/collection1/lib/README' to classloader
[junit4:junit4] 2> 596942 T2168 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 596988 T2168 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 597038 T2168 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 597039 T2168 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 597044 T2168 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 597605 T2168 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 597611 T2168 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 597614 T2168 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 597624 T2168 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 597628 T2168 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 597630 T2168 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 597631 T2168 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 597632 T2168 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 597632 T2168 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 597633 T2168 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 597633 T2168 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 597633 T2168 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 597633 T2168 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371349530227/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/
[junit4:junit4] 2> 597634 T2168 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11ef443
[junit4:junit4] 2> 597634 T2168 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 597635 T2168 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6
[junit4:junit4] 2> 597635 T2168 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index/
[junit4:junit4] 2> 597635 T2168 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 597635 T2168 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index
[junit4:junit4] 2> 597639 T2168 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 597639 T2168 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 597641 T2168 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 597641 T2168 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 597641 T2168 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 597642 T2168 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 597643 T2168 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 597643 T2168 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 597643 T2168 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 597644 T2168 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 597644 T2168 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 597646 T2168 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 597649 T2168 oass.SolrIndexSearcher.<init> Opening Searcher@6f2f22 main
[junit4:junit4] 2> 597650 T2168 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 597650 T2168 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 597655 T2169 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6f2f22 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 597662 T2168 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 597663 T2168 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49290 collection:collection1 shard:shard3
[junit4:junit4] 2> 597665 T2168 oasc.ZkController.register We are http://127.0.0.1:49290/collection1/ and leader is http://127.0.0.1:49705/collection1/
[junit4:junit4] 2> 597665 T2168 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49290
[junit4:junit4] 2> 597666 T2168 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 597666 T2168 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C3878 name=collection1 org.apache.solr.core.SolrCore@1550cb9 url=http://127.0.0.1:49290/collection1 node=127.0.0.1:49290_ C3878_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49290_, base_url=http://127.0.0.1:49290}
[junit4:junit4] 2> 597666 T2170 C3878 P49290 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 597666 T2168 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 597667 T2170 C3878 P49290 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 597667 T2170 C3878 P49290 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 597668 T2170 C3878 P49290 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 597668 T2041 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
[junit4:junit4] 2> 597668 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 597668 T2170 C3878 P49290 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 597669 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 597673 T2111 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> ASYNC NEW_CORE C3879 name=collection1 org.apache.solr.core.SolrCore@1b672fa url=http://127.0.0.1:59158/collection1 node=127.0.0.1:59158_ C3879_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:59158_, base_url=http://127.0.0.1:59158}
[junit4:junit4] 2> 597699 T2151 C3879 P59158 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:57342/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 597699 T2151 C3879 P59158 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59158 START replicas=[http://127.0.0.1:57342/collection1/] nUpdates=100
[junit4:junit4] 2> 597699 T2151 C3879 P59158 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 597700 T2151 C3879 P59158 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 597700 T2151 C3879 P59158 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 597701 T2151 C3879 P59158 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 597701 T2151 C3879 P59158 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 597701 T2151 C3879 P59158 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:57342/collection1/. core=collection1
[junit4:junit4] 2> 597701 T2151 C3879 P59158 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3880 name=collection1 org.apache.solr.core.SolrCore@9995b7 url=http://127.0.0.1:57342/collection1 node=127.0.0.1:57342_ C3880_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:57342_, base_url=http://127.0.0.1:57342, leader=true}
[junit4:junit4] 2> 597708 T2092 C3880 P57342 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 597709 T2094 C3880 P57342 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 597711 T2094 C3880 P57342 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 597711 T2094 C3880 P57342 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 597717 T2094 C3880 P57342 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 597717 T2094 C3880 P57342 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 597718 T2094 C3880 P57342 oass.SolrIndexSearcher.<init> Opening Searcher@166af11 realtime
[junit4:junit4] 2> 597719 T2094 C3880 P57342 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 597719 T2094 C3880 P57342 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4] 2> 597720 T2151 C3879 P59158 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 597720 T2151 C3879 P59158 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 597721 T2095 C3880 P57342 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 597722 T2095 C3880 P57342 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 597722 T2151 C3879 P59158 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 597722 T2151 C3879 P59158 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 597723 T2151 C3879 P59158 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 597724 T2092 C3880 P57342 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 597725 T2151 C3879 P59158 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 597726 T2151 C3879 P59158 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index.20130616052533255
[junit4:junit4] 2> 597726 T2151 C3879 P59158 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index.20130616052533255 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d35d6b fullCopy=false
[junit4:junit4] 2> 597728 T2094 C3880 P57342 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 597729 T2151 C3879 P59158 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 597731 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 597732 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49290_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49290"}
[junit4:junit4] 2> 597747 T2151 C3879 P59158 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 597747 T2151 C3879 P59158 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 597749 T2151 C3879 P59158 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 597749 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 597750 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 597749 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 597750 T2151 C3879 P59158 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 597749 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 597749 T2165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 597749 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 597749 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 597749 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 597751 T2151 C3879 P59158 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 597753 T2151 C3879 P59158 oass.SolrIndexSearcher.<init> Opening Searcher@fa8e08 main
[junit4:junit4] 2> 597754 T2150 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@fa8e08 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 597755 T2151 C3879 P59158 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index.20130616052533255 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index.20130616052533255;done=true>>]
[junit4:junit4] 2> 597755 T2151 C3879 P59158 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index.20130616052533255
[junit4:junit4] 2> 597755 T2151 C3879 P59158 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty5/index.20130616052533255
[junit4:junit4] 2> 597756 T2151 C3879 P59158 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 597756 T2151 C3879 P59158 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 597756 T2151 C3879 P59158 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 597756 T2151 C3879 P59158 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 597757 T2151 C3879 P59158 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 597764 T2041 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 597770 T2041 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33622
[junit4:junit4] 2> 597771 T2041 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 597771 T2041 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 597772 T2041 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371349533202
[junit4:junit4] 2> 597772 T2041 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371349533202/solr.xml
[junit4:junit4] 2> 597773 T2041 oasc.CoreContainer.<init> New CoreContainer 7921260
[junit4:junit4] 2> 597773 T2041 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371349533202/'
[junit4:junit4] 2> 597774 T2041 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371349533202/'
[junit4:junit4] 2> 597854 T2041 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 597855 T2041 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 597855 T2041 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 597855 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 597856 T2041 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 597856 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 597857 T2041 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 597857 T2041 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 597857 T2041 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 597858 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 597861 T2041 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 597862 T2041 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54854/solr
[junit4:junit4] 2> 597862 T2041 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 597864 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 597866 T2184 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@b7755f name:ZooKeeperConnection Watcher:127.0.0.1:54854 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 597866 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 597867 T2041 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 597872 T2041 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 597873 T2186 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b7483 name:ZooKeeperConnection Watcher:127.0.0.1:54854/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 597873 T2041 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 597876 T2041 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 598673 T2111 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 598674 T2111 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:49290_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 598878 T2041 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33622_
[junit4:junit4] 2> 598880 T2041 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33622_
[junit4:junit4] 2> 598882 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 598882 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 598882 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 598883 T2165 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598882 T2131 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598883 T2063 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598883 T2165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 598883 T2083 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598883 T2186 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598883 T2148 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598884 T2070 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598884 T2115 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598884 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 598884 T2099 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 598883 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 598883 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 598885 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 598889 T2187 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 598890 T2187 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 599255 T2064 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 599256 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59158_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59158"}
[junit4:junit4] 2> 599259 T2064 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:33622_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:33622"}
[junit4:junit4] 2> 599264 T2064 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 599264 T2064 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 599269 T2099 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599269 T2165 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599269 T2148 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599270 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599269 T2186 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599269 T2070 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599269 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599269 T2131 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599274 T2063 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 599891 T2187 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371349533202/collection1
[junit4:junit4] 2> 599891 T2187 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 599894 T2187 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 599894 T2187 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 599895 T2187 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371349533202/collection1/'
[junit4:junit4] 2> 599896 T2187 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371349533202/collection1/lib/README' to classloader
[junit4:junit4] 2> 599897 T2187 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371349533202/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 599957 T2187 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 600043 T2187 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 600046 T2187 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 600053 T2187 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> ASYNC NEW_CORE C3881 name=collection1 org.apache.solr.core.SolrCore@1550cb9 url=http://127.0.0.1:49290/collection1 node=127.0.0.1:49290_ C3881_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49290_, base_url=http://127.0.0.1:49290}
[junit4:junit4] 2> 600675 T2170 C3881 P49290 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49705/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 600675 T2170 C3881 P49290 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:49290 START replicas=[http://127.0.0.1:49705/collection1/] nUpdates=100
[junit4:junit4] 2> 600677 T2170 C3881 P49290 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 600678 T2170 C3881 P49290 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 600678 T2170 C3881 P49290 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 600678 T2170 C3881 P49290 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 600678 T2170 C3881 P49290 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 600679 T2170 C3881 P49290 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49705/collection1/. core=collection1
[junit4:junit4] 2> 600679 T2170 C3881 P49290 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C3882 name=collection1 org.apache.solr.core.SolrCore@1cf7abb url=http://127.0.0.1:49705/collection1 node=127.0.0.1:49705_ C3882_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49705_, base_url=http://127.0.0.1:49705, leader=true}
[junit4:junit4] 2> 600685 T2110 C3882 P49705 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> ASYNC NEW_CORE C3883 name=collection1 org.apache.solr.core.SolrCore@1cf7abb url=http://127.0.0.1:49705/collection1 node=127.0.0.1:49705_ C3883_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49705_, base_url=http://127.0.0.1:49705, leader=true}
[junit4:junit4] 2> 600697 T2109 C3883 P49705 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 600706 T2109 C3883 P49705 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 600707 T2109 C3883 P49705 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 600717 T2109 C3883 P49705 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty3/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 600718 T2109 C3883 P49705 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 600719 T2109 C3883 P49705 oass.SolrIndexSearcher.<init> Opening Searcher@be387b realtime
[junit4:junit4] 2> 600719 T2109 C3883 P49705 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 600719 T2109 C3883 P49705 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 22
[junit4:junit4] 2> ASYNC NEW_CORE C3884 name=collection1 org.apache.solr.core.SolrCore@1550cb9 url=http://127.0.0.1:49290/collection1 node=127.0.0.1:49290_ C3884_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49290_, base_url=http://127.0.0.1:49290}
[junit4:junit4] 2> 600720 T2170 C3884 P49290 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 600720 T2170 C3884 P49290 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 600722 T2111 C3883 P49705 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 600723 T2111 C3883 P49705 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4] 2> 600724 T2170 C3884 P49290 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 600724 T2170 C3884 P49290 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 600724 T2170 C3884 P49290 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 600727 T2108 C3883 P49705 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4] 2> 600728 T2170 C3884 P49290 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 600731 T2170 C3884 P49290 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index.20130616052536259
[junit4:junit4] 2> 600731 T2170 C3884 P49290 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.NIOFSDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index.20130616052536259 lockFactory=org.apache.lucene.store.NativeFSLockFactory@e78fc9 fullCopy=false
[junit4:junit4] 2> ASYNC NEW_CORE C3885 name=collection1 org.apache.solr.core.SolrCore@1cf7abb url=http://127.0.0.1:49705/collection1 node=127.0.0.1:49705_ C3885_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:49705_, base_url=http://127.0.0.1:49705, leader=true}
[junit4:junit4] 2> 600740 T2110 C3885 P49705 oasc.SolrCore.execute [collection1] webapp= path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> ASYNC NEW_CORE C3886 name=collection1 org.apache.solr.core.SolrCore@1550cb9 url=http://127.0.0.1:49290/collection1 node=127.0.0.1:49290_ C3886_STATE=coll:collection1 core:collection1 props:{shard=shard3, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:49290_, base_url=http://127.0.0.1:49290}
[junit4:junit4] 2> 600742 T2170 C3886 P49290 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 600757 T2170 C3886 P49290 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 600758 T2170 C3886 P49290 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 600760 T2170 C3886 P49290 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 600761 T2170 C3886 P49290 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 600762 T2170 C3886 P49290 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 600762 T2170 C3886 P49290 oass.SolrIndexSearcher.<init> Opening Searcher@1a83ecb main
[junit4:junit4] 2> 600764 T2169 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a83ecb main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 600764 T2170 C3886 P49290 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index.20130616052536259 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index.20130616052536259;done=true>>]
[junit4:junit4] 2> 600765 T2170 C3886 P49290 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index.20130616052536259
[junit4:junit4] 2> 600765 T2170 C3886 P49290 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty6/index.20130616052536259
[junit4:junit4] 2> 600765 T2170 C3886 P49290 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 600765 T2170 C3886 P49290 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 600766 T2170 C3886 P49290 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 600766 T2170 C3886 P49290 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 600
[...truncated too long message...]
eader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 744908 T2523 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"12",
[junit4:junit4] 2> "numShards":null,
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:54022_",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:54022"}
[junit4:junit4] 2> 744911 T2442 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 744919 T2041 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 744971 T2041 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 34389
[junit4:junit4] 2> 744973 T2041 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=32141092
[junit4:junit4] 2> 745105 T2461 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 745105 T2461 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 745106 T2461 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 745975 T2041 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 745976 T2041 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 745979 T2041 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1513b29
[junit4:junit4] 2> 745996 T2041 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=72,cumulative_deletesById=37,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4] 2> 745997 T2041 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 745997 T2041 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 745997 T2041 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 745998 T2041 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 745999 T2041 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 745999 T2041 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty12/index.20130616052715959 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty12/index.20130616052715959;done=false>>]
[junit4:junit4] 2> 745999 T2041 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty12/index.20130616052715959
[junit4:junit4] 2> 746000 T2041 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty12;done=false>>]
[junit4:junit4] 2> 746000 T2041 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371349508364/jetty12
[junit4:junit4] 2> 746001 T2523 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89872761380470826-127.0.0.1:34389_-n_0000000019) am no longer a leader.
[junit4:junit4] 2> 746024 T2041 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4] 2> 746110 T2041 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 746111 T2041 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:54854 54854
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=47A3AE6B36F45749 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr -Dtests.timezone=Africa/Asmera -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 173s J0 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard1 is not consistent. Got 227 from http://127.0.0.1:33622/collection1lastClient and got 237 from http://127.0.0.1:57036/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([47A3AE6B36F45749:C645207341AB3775]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:662)
[junit4:junit4] 2> 746126 T2041 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 173303 T2040 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4] 2> 747302 T2442 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 747302 T2442 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 747303 T2442 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=sr, timezone=Africa/Asmera
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=3,free=186114224,total=409370624
[junit4:junit4] 2> NOTE: All tests run in this JVM: [SpellingQueryConverterTest, TestFunctionQuery, DistributedSpellCheckComponentTest, LegacyHTMLStripCharFilterTest, TermsComponentTest, TestConfig, TestRemoteStreaming, SliceStateTest, SuggesterFSTTest, AnalysisAfterCoreReloadTest, OverseerTest, PluginInfoTest, SuggesterTest, CoreAdminHandlerTest, AliasIntegrationTest, SyncSliceTest, AutoCommitTest, SoftAutoCommitTest, BadCopyFieldTest, PrimitiveFieldTypeTest, TestSurroundQueryParser, TestLFUCache, BasicDistributedZkTest, TestDocumentBuilder, StatelessScriptUpdateProcessorFactoryTest, ExternalFileFieldSortTest, TestJoin, CachingDirectoryFactoryTest, TestGroupingSearch, RAMDirectoryFactoryTest, DistributedQueryElevationComponentTest, TestFieldCollectionResource, TestSchemaResource, LukeRequestHandlerTest, TestLMDirichletSimilarityFactory, TestSolrCoreProperties, TestQuerySenderListener, TestQueryUtils, BinaryUpdateRequestHandlerTest, PreAnalyzedUpdateProcessorTest, ReturnFieldsTest, ResponseLogComponentTest, TestPseudoReturnFields, AlternateDirectoryTest, JsonLoaderTest, ChaosMonkeyNothingIsSafeTest, CircularListTest, FastVectorHighlighterTest, DateFieldTest, TestSolrIndexConfig, PrimUtilsTest, SimpleFacetsTest, TestPropInjectDefaults, QueryResultKeyTest, TestSolrDeletionPolicy1, TestLRUCache, TestSchemaSimilarityResource, TestTrie, PolyFieldTest, TestPHPSerializedResponseWriter, SystemInfoHandlerTest, FileBasedSpellCheckerTest, RegexBoostProcessorTest, TestMaxScoreQueryParser, LeaderElectionTest, TestSort, ResourceLoaderTest, PeerSyncTest, TestSearchPerf, TestCloudManagedSchema, TestSolrDeletionPolicy2, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J0 in 174.76s, 1 test, 1 failure <<< FAILURES!
[...truncated 460 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 297 suites, 1246 tests, 1 failure, 13 ignored (7 assumptions)
Total time: 45 minutes 6 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure