You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/17 16:06:03 UTC
[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_21) - Build # 6105
- Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6105/
Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard3 is not consistent. Got 648 from http://127.0.0.1:49202/_xpo/ip/collection1lastClient and got 646 from http://127.0.0.1:37443/_xpo/ip/collection1
Stack Trace:
java.lang.AssertionError: shard3 is not consistent. Got 648 from http://127.0.0.1:49202/_xpo/ip/collection1lastClient and got 646 from http://127.0.0.1:37443/_xpo/ip/collection1
at __randomizedtesting.SeedInfo.seed([62718A172099C606:E397040F57C6A63A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:722)
Build Log:
[...truncated 9329 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4] 2> 93216 T217 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_xpo/ip
[junit4:junit4] 2> 93233 T217 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371476825183
[junit4:junit4] 2> 93238 T217 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4] 2> 93241 T218 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4] 2> 93341 T217 oasc.ZkTestServer.run start zk server on port:35312
[junit4:junit4] 2> 93386 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 93537 T224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3756a51e name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 93537 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 93539 T217 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4] 2> 93576 T219 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13f526394b40000, likely client has closed socket
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
[junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2>
[junit4:junit4] 2> 93580 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 93582 T226 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@636c241c name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 93583 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 93585 T217 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4] 2> 93591 T217 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4] 2> 93599 T217 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4] 2> 93602 T217 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4] 2> 93607 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 93608 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4] 2> 93615 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4] 2> 93615 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4] 2> 93620 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4] 2> 93621 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4] 2> 93626 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4] 2> 93626 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4] 2> 93630 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4] 2> 93631 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4] 2> 93644 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 93645 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4] 2> 93649 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 93650 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4] 2> 93654 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 93655 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4] 2> 93659 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4] 2> 93660 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4] 2> 93751 T217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 93758 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53862
[junit4:junit4] 2> 93759 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 93759 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 93759 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617
[junit4:junit4] 2> 93760 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/solr.xml
[junit4:junit4] 2> 93760 T217 oasc.CoreContainer.<init> New CoreContainer 1639190412
[junit4:junit4] 2> 93760 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/'
[junit4:junit4] 2> 93761 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/'
[junit4:junit4] 2> 93806 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 93807 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 93807 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 93807 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 93808 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 93808 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 93808 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 93809 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 93809 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 93809 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 93862 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 93862 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
[junit4:junit4] 2> 93868 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 93869 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 93871 T237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19e0ddff name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 93872 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 93876 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 93882 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 93883 T239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a4360d5 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 93884 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 93889 T217 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4] 2> 93897 T217 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4] 2> 93905 T217 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4] 2> 93908 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53862__xpo%2Fip
[junit4:junit4] 2> 93911 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53862__xpo%2Fip
[junit4:junit4] 2> 93916 T217 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4] 2> 93928 T217 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4] 2> 93932 T217 oasc.Overseer.start Overseer (id=89881105220763651-127.0.0.1:53862__xpo%2Fip-n_0000000000) starting
[junit4:junit4] 2> 93948 T217 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4] 2> 93963 T241 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4] 2> 93963 T217 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4] 2> 93968 T217 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4] 2> 93972 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 93984 T240 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4] 2> 93992 T242 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 93993 T242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 95491 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 95493 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53862__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53862/_xpo/ip"}
[junit4:junit4] 2> 95493 T240 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4] 2> 95504 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 95520 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 95995 T242 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1
[junit4:junit4] 2> 95996 T242 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4] 2> 95997 T242 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 95997 T242 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4] 2> 96000 T242 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1/'
[junit4:junit4] 2> 96001 T242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1/lib/README' to classloader
[junit4:junit4] 2> 96001 T242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 96049 T242 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 96091 T242 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 96094 T242 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 96115 T242 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 96490 T242 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 96498 T242 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 96501 T242 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 96528 T242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 96533 T242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 96538 T242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 96540 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 96540 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 96540 T242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 96542 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 96542 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 96542 T242 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 96549 T242 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/
[junit4:junit4] 2> 96549 T242 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
[junit4:junit4] 2> 96550 T242 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 96550 T242 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data
[junit4:junit4] 2> 96551 T242 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/index/
[junit4:junit4] 2> 96551 T242 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 96551 T242 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/index
[junit4:junit4] 2> 96555 T242 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 96555 T242 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 96557 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 96558 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 96559 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 96559 T242 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 96560 T242 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 96560 T242 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 96560 T242 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 96560 T242 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 96561 T242 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 96563 T242 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 96566 T242 oass.SolrIndexSearcher.<init> Opening Searcher@69665ade main
[junit4:junit4] 2> 96567 T242 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 96567 T242 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 96571 T243 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69665ade main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 96573 T242 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 96573 T242 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53862/_xpo/ip collection:control_collection shard:shard1
[junit4:junit4] 2> 96578 T242 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 96586 T242 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4] 2> 96599 T242 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 96603 T242 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 96604 T242 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 96605 T242 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53862/_xpo/ip/collection1/
[junit4:junit4] 2> 96606 T242 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 96606 T242 oasc.SyncStrategy.syncToMe http://127.0.0.1:53862/_xpo/ip/collection1/ has no replicas
[junit4:junit4] 2> 96607 T242 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53862/_xpo/ip/collection1/
[junit4:junit4] 2> 96607 T242 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4] 2> 97028 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 97039 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 97069 T242 oasc.ZkController.register We are http://127.0.0.1:53862/_xpo/ip/collection1/ and leader is http://127.0.0.1:53862/_xpo/ip/collection1/
[junit4:junit4] 2> 97069 T242 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53862/_xpo/ip
[junit4:junit4] 2> 97069 T242 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 97069 T242 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 97070 T242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 97072 T242 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 97074 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 97074 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 97088 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 97094 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4] 2> 97096 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 97098 T246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cd647e3 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 97098 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 97100 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 97104 T217 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4] 2> 97168 T217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 97170 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55519
[junit4:junit4] 2> 97170 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 97171 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 97171 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053
[junit4:junit4] 2> 97171 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/solr.xml
[junit4:junit4] 2> 97171 T217 oasc.CoreContainer.<init> New CoreContainer 849998145
[junit4:junit4] 2> 97172 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/'
[junit4:junit4] 2> 97172 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/'
[junit4:junit4] 2> 97226 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 97227 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 97227 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 97227 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 97228 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 97228 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 97228 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 97229 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 97229 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 97230 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 97236 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 97237 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
[junit4:junit4] 2> 97237 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 97238 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 97240 T257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@478bb673 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 97241 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 97243 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 97250 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 97252 T259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@406993a1 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 97252 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 97258 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 98262 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55519__xpo%2Fip
[junit4:junit4] 2> 98264 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55519__xpo%2Fip
[junit4:junit4] 2> 98266 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 98268 T246 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 98268 T259 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 98269 T239 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4] 2> 98276 T260 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 98276 T260 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 98546 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 98547 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"1",
[junit4:junit4] 2> "numShards":"1",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"control_collection",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:53862__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:53862/_xpo/ip"}
[junit4:junit4] 2> 98552 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55519__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55519/_xpo/ip"}
[junit4:junit4] 2> 98552 T240 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4] 2> 98552 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 98558 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 98558 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 98558 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 99278 T260 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1
[junit4:junit4] 2> 99278 T260 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 99279 T260 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 99280 T260 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 99282 T260 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1/'
[junit4:junit4] 2> 99282 T260 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1/lib/README' to classloader
[junit4:junit4] 2> 99283 T260 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 99310 T260 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 99360 T260 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 99363 T260 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 99385 T260 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 99717 T260 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 99728 T260 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 99731 T260 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 99761 T260 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 99766 T260 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 99771 T260 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 99773 T260 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 99774 T260 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 99774 T260 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 99776 T260 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 99776 T260 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 99776 T260 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 99777 T260 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/
[junit4:junit4] 2> 99777 T260 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
[junit4:junit4] 2> 99777 T260 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 99778 T260 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1
[junit4:junit4] 2> 99778 T260 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index/
[junit4:junit4] 2> 99779 T260 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 99779 T260 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index
[junit4:junit4] 2> 99782 T260 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 99782 T260 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 99784 T260 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 99785 T260 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 99785 T260 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 99786 T260 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 99786 T260 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 99787 T260 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 99787 T260 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 99787 T260 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 99788 T260 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 99790 T260 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 99795 T260 oass.SolrIndexSearcher.<init> Opening Searcher@254c25e1 main
[junit4:junit4] 2> 99796 T260 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 99796 T260 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 99802 T260 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 99802 T260 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55519/_xpo/ip collection:collection1 shard:shard1
[junit4:junit4] 2> 99803 T260 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4] 2> 99804 T261 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@254c25e1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 99812 T260 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 99817 T260 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 99818 T260 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 99818 T260 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55519/_xpo/ip/collection1/
[junit4:junit4] 2> 99818 T260 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 99818 T260 oasc.SyncStrategy.syncToMe http://127.0.0.1:55519/_xpo/ip/collection1/ has no replicas
[junit4:junit4] 2> 99819 T260 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55519/_xpo/ip/collection1/
[junit4:junit4] 2> 99819 T260 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4] 2> 100067 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 100155 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 100155 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 100155 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 100177 T260 oasc.ZkController.register We are http://127.0.0.1:55519/_xpo/ip/collection1/ and leader is http://127.0.0.1:55519/_xpo/ip/collection1/
[junit4:junit4] 2> 100178 T260 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55519/_xpo/ip
[junit4:junit4] 2> 100178 T260 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 100178 T260 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 100178 T260 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 100181 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 100182 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 100183 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 100183 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 100250 T217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 100252 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58042
[junit4:junit4] 2> 100253 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 100254 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 100254 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136
[junit4:junit4] 2> 100254 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/solr.xml
[junit4:junit4] 2> 100254 T217 oasc.CoreContainer.<init> New CoreContainer 1776184614
[junit4:junit4] 2> 100255 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/'
[junit4:junit4] 2> 100255 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/'
[junit4:junit4] 2> 100319 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 100320 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 100320 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 100321 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 100321 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 100322 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 100322 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 100323 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 100323 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 100323 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 100329 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 100330 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
[junit4:junit4] 2> 100331 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 100332 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 100334 T273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66161e13 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 100334 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 100338 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 100343 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 100347 T275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bc8ef7b name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 100347 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 100360 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 101364 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58042__xpo%2Fip
[junit4:junit4] 2> 101366 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58042__xpo%2Fip
[junit4:junit4] 2> 101368 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 101369 T246 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 101370 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 101372 T259 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 101372 T239 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 101372 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 101372 T275 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4] 2> 101381 T276 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 101381 T276 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 101666 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 101667 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"2",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:55519__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:55519/_xpo/ip"}
[junit4:junit4] 2> 101671 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:58042__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:58042/_xpo/ip"}
[junit4:junit4] 2> 101672 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 101672 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 101679 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 101679 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 101679 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 101679 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 102383 T276 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1
[junit4:junit4] 2> 102383 T276 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 102384 T276 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 102385 T276 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 102387 T276 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1/'
[junit4:junit4] 2> 102388 T276 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1/lib/README' to classloader
[junit4:junit4] 2> 102388 T276 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 102426 T276 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 102464 T276 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 102466 T276 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 102482 T276 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 102878 T276 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 102886 T276 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 102888 T276 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 102913 T276 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 102917 T276 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 102923 T276 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 102925 T276 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 102925 T276 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 102928 T276 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 102930 T276 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 102930 T276 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 102930 T276 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 102931 T276 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/
[junit4:junit4] 2> 102931 T276 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
[junit4:junit4] 2> 102931 T276 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 102932 T276 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2
[junit4:junit4] 2> 102932 T276 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index/
[junit4:junit4] 2> 102932 T276 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 102933 T276 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index
[junit4:junit4] 2> 102936 T276 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 102937 T276 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 102940 T276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 102940 T276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 102941 T276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 102941 T276 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 102942 T276 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 102942 T276 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 102943 T276 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 102943 T276 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 102944 T276 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 102946 T276 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 102949 T276 oass.SolrIndexSearcher.<init> Opening Searcher@7e74b92b main
[junit4:junit4] 2> 102950 T276 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 102950 T276 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 102954 T277 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e74b92b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 102956 T276 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 102957 T276 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58042/_xpo/ip collection:collection1 shard:shard2
[junit4:junit4] 2> 102958 T276 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4] 2> 102967 T276 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 102972 T276 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 102972 T276 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 102972 T276 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58042/_xpo/ip/collection1/
[junit4:junit4] 2> 102973 T276 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 102973 T276 oasc.SyncStrategy.syncToMe http://127.0.0.1:58042/_xpo/ip/collection1/ has no replicas
[junit4:junit4] 2> 102973 T276 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58042/_xpo/ip/collection1/
[junit4:junit4] 2> 102973 T276 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4] 2> 103192 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 103203 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 103203 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 103203 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 103203 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 103230 T276 oasc.ZkController.register We are http://127.0.0.1:58042/_xpo/ip/collection1/ and leader is http://127.0.0.1:58042/_xpo/ip/collection1/
[junit4:junit4] 2> 103231 T276 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58042/_xpo/ip
[junit4:junit4] 2> 103231 T276 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 103231 T276 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 103231 T276 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 103234 T276 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 103236 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 103236 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 103237 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 103337 T217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 103340 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44496
[junit4:junit4] 2> 103341 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 103341 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 103342 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191
[junit4:junit4] 2> 103343 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/solr.xml
[junit4:junit4] 2> 103343 T217 oasc.CoreContainer.<init> New CoreContainer 248526691
[junit4:junit4] 2> 103344 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/'
[junit4:junit4] 2> 103344 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/'
[junit4:junit4] 2> 103411 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 103411 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 103411 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 103412 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 103412 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 103412 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 103413 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 103413 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 103413 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 103414 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 103418 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 103418 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
[junit4:junit4] 2> 103419 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 103420 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 103422 T289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e3aae18 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 103422 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 103426 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 103432 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 103436 T291 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8cd9e1 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 103437 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 103443 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 104448 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44496__xpo%2Fip
[junit4:junit4] 2> 104449 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44496__xpo%2Fip
[junit4:junit4] 2> 104452 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 104454 T246 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 104454 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 104454 T275 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 104454 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 104454 T259 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 104455 T291 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 104455 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 104456 T239 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4] 2> 104463 T292 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 104463 T292 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 104712 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 104713 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"3",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:58042__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:58042/_xpo/ip"}
[junit4:junit4] 2> 104718 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44496__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44496/_xpo/ip"}
[junit4:junit4] 2> 104718 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 104718 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 104724 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 104724 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 104725 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 104724 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 104724 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 105465 T292 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1
[junit4:junit4] 2> 105465 T292 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 105467 T292 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 105468 T292 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 105471 T292 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1/'
[junit4:junit4] 2> 105472 T292 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1/lib/README' to classloader
[junit4:junit4] 2> 105473 T292 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 105509 T292 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 105546 T292 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 105547 T292 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 105556 T292 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 105925 T292 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 105934 T292 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 105938 T292 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 105964 T292 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 105968 T292 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 105972 T292 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 105974 T292 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 105975 T292 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 105975 T292 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 105976 T292 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 105976 T292 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 105977 T292 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 105977 T292 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/
[junit4:junit4] 2> 105977 T292 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
[junit4:junit4] 2> 105978 T292 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 105978 T292 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3
[junit4:junit4] 2> 105978 T292 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/index/
[junit4:junit4] 2> 105979 T292 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 105979 T292 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/index
[junit4:junit4] 2> 105983 T292 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 105983 T292 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 105986 T292 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 105987 T292 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 105987 T292 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 105988 T292 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 105989 T292 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 105989 T292 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 105989 T292 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 105990 T292 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 105991 T292 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 105995 T292 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 105999 T292 oass.SolrIndexSearcher.<init> Opening Searcher@7e9dbe8a main
[junit4:junit4] 2> 106000 T292 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 106000 T292 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 106006 T293 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e9dbe8a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 106008 T292 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 106008 T292 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44496/_xpo/ip collection:collection1 shard:shard3
[junit4:junit4] 2> 106009 T292 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4] 2> 106017 T292 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4] 2> 106021 T292 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4] 2> 106021 T292 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4] 2> 106021 T292 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44496/_xpo/ip/collection1/
[junit4:junit4] 2> 106021 T292 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4] 2> 106022 T292 oasc.SyncStrategy.syncToMe http://127.0.0.1:44496/_xpo/ip/collection1/ has no replicas
[junit4:junit4] 2> 106022 T292 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44496/_xpo/ip/collection1/
[junit4:junit4] 2> 106022 T292 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4] 2> 106238 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 106248 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106248 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106248 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106249 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106252 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 106280 T292 oasc.ZkController.register We are http://127.0.0.1:44496/_xpo/ip/collection1/ and leader is http://127.0.0.1:44496/_xpo/ip/collection1/
[junit4:junit4] 2> 106280 T292 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44496/_xpo/ip
[junit4:junit4] 2> 106280 T292 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4] 2> 106281 T292 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 106281 T292 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 106284 T292 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 106285 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 106286 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 106286 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 106375 T217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 106378 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59777
[junit4:junit4] 2> 106379 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 106380 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 106380 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239
[junit4:junit4] 2> 106380 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/solr.xml
[junit4:junit4] 2> 106381 T217 oasc.CoreContainer.<init> New CoreContainer 1160373085
[junit4:junit4] 2> 106381 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/'
[junit4:junit4] 2> 106381 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/'
[junit4:junit4] 2> 106439 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 106440 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 106440 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 106441 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 106441 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 106441 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 106442 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 106442 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 106442 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 106443 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 106446 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 106447 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
[junit4:junit4] 2> 106447 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 106448 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 106450 T305 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@136cf name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 106451 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 106453 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 106457 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 106459 T307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6415cf5a name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 106459 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 106464 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 107468 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59777__xpo%2Fip
[junit4:junit4] 2> 107470 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59777__xpo%2Fip
[junit4:junit4] 2> 107473 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4] 2> 107475 T307 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 107475 T259 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 107476 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107476 T291 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 107477 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107476 T246 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 107476 T275 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 107478 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107478 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107479 T239 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4] 2> 107486 T308 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 107486 T308 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 107761 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 107762 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"4",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:44496__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:44496/_xpo/ip"}
[junit4:junit4] 2> 107767 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59777__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59777/_xpo/ip"}
[junit4:junit4] 2> 107768 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 107768 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 107835 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107835 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107835 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107835 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107835 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 107835 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 108488 T308 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1
[junit4:junit4] 2> 108489 T308 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 108490 T308 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 108490 T308 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 108491 T308 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1/'
[junit4:junit4] 2> 108492 T308 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1/lib/README' to classloader
[junit4:junit4] 2> 108493 T308 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 108532 T308 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 108569 T308 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 108571 T308 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 108578 T308 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 108907 T308 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 108913 T308 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 108916 T308 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 108937 T308 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 108942 T308 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 108949 T308 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 108951 T308 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 108951 T308 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 108951 T308 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 108953 T308 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 108953 T308 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 108954 T308 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 108954 T308 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/
[junit4:junit4] 2> 108954 T308 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
[junit4:junit4] 2> 108955 T308 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 108955 T308 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4
[junit4:junit4] 2> 108955 T308 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index/
[junit4:junit4] 2> 108956 T308 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 108956 T308 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index
[junit4:junit4] 2> 109026 T308 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 109026 T308 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 109029 T308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 109030 T308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 109030 T308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 109031 T308 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 109032 T308 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 109032 T308 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 109033 T308 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 109033 T308 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 109034 T308 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 109037 T308 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 109041 T308 oass.SolrIndexSearcher.<init> Opening Searcher@70c8ea27 main
[junit4:junit4] 2> 109042 T308 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 109043 T308 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 109047 T309 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@70c8ea27 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 109050 T308 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 109050 T308 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59777/_xpo/ip collection:collection1 shard:shard1
[junit4:junit4] 2> 109054 T308 oasc.ZkController.register We are http://127.0.0.1:59777/_xpo/ip/collection1/ and leader is http://127.0.0.1:55519/_xpo/ip/collection1/
[junit4:junit4] 2> 109054 T308 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59777/_xpo/ip
[junit4:junit4] 2> 109054 T308 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 109054 T308 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C210 name=collection1 org.apache.solr.core.SolrCore@13931c0f url=http://127.0.0.1:59777/_xpo/ip/collection1 node=127.0.0.1:59777__xpo%2Fip C210_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:59777__xpo%2Fip, base_url=http://127.0.0.1:59777/_xpo/ip}
[junit4:junit4] 2> 109058 T310 C210 P59777 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 109059 T310 C210 P59777 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 109059 T308 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 109059 T310 C210 P59777 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 109059 T310 C210 P59777 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 109060 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 109060 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 109061 T310 C210 P59777 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 109061 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 109164 T217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 109166 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37170
[junit4:junit4] 2> 109167 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 109167 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 109168 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015
[junit4:junit4] 2> 109168 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/solr.xml
[junit4:junit4] 2> 109169 T217 oasc.CoreContainer.<init> New CoreContainer 175101158
[junit4:junit4] 2> 109169 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/'
[junit4:junit4] 2> 109170 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/'
[junit4:junit4] 2> 109222 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 109222 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 109223 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 109223 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 109223 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 109224 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 109224 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 109224 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 109225 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 109225 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 109229 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 109229 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
[junit4:junit4] 2> 109230 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 109230 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 109232 T322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75e9584 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 109232 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 109234 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 109238 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 109239 T324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19b07bac name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 109239 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 109244 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> 109284 T254 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 109351 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 109353 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59777__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59777/_xpo/ip"}
[junit4:junit4] 2> 109359 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109360 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109360 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109360 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109360 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109359 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 109359 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110248 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37170__xpo%2Fip
[junit4:junit4] 2> 110249 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37170__xpo%2Fip
[junit4:junit4] 2> 110253 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110253 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4] 2> 110254 T307 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 110254 T291 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 110255 T259 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 110254 T275 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 110254 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110254 T246 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 110255 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110255 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110255 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110257 T324 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 110257 T239 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4] 2> 110256 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110264 T325 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 110264 T325 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 110286 T254 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 110286 T254 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:59777__xpo%252Fip&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
[junit4:junit4] 2> 110865 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 110867 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37170__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37170/_xpo/ip"}
[junit4:junit4] 2> 110868 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 110869 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4] 2> 110876 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110877 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110877 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110877 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110876 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110876 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 110876 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 111266 T325 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1
[junit4:junit4] 2> 111267 T325 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 111268 T325 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 111269 T325 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 111271 T325 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1/'
[junit4:junit4] 2> 111273 T325 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1/lib/README' to classloader
[junit4:junit4] 2> 111273 T325 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 111317 T325 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 111352 T325 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 111354 T325 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 111361 T325 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 111705 T325 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 111712 T325 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 111714 T325 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 111736 T325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 111740 T325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 111743 T325 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 111744 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 111744 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 111745 T325 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 111746 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 111746 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 111746 T325 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 111746 T325 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/
[junit4:junit4] 2> 111747 T325 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
[junit4:junit4] 2> 111747 T325 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 111748 T325 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5
[junit4:junit4] 2> 111748 T325 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index/
[junit4:junit4] 2> 111748 T325 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 111748 T325 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index
[junit4:junit4] 2> 111751 T325 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 111751 T325 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 111758 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 111759 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 111759 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 111760 T325 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 111760 T325 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 111761 T325 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 111761 T325 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 111761 T325 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 111762 T325 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 111764 T325 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 111767 T325 oass.SolrIndexSearcher.<init> Opening Searcher@2a9ecf43 main
[junit4:junit4] 2> 111768 T325 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 111768 T325 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 111773 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2a9ecf43 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 111775 T325 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 111775 T325 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37170/_xpo/ip collection:collection1 shard:shard2
[junit4:junit4] 2> 111778 T325 oasc.ZkController.register We are http://127.0.0.1:37170/_xpo/ip/collection1/ and leader is http://127.0.0.1:58042/_xpo/ip/collection1/
[junit4:junit4] 2> 111778 T325 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37170/_xpo/ip
[junit4:junit4] 2> 111778 T325 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 111779 T325 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C211 name=collection1 org.apache.solr.core.SolrCore@a0a3409 url=http://127.0.0.1:37170/_xpo/ip/collection1 node=127.0.0.1:37170__xpo%2Fip C211_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:37170__xpo%2Fip, base_url=http://127.0.0.1:37170/_xpo/ip}
[junit4:junit4] 2> 111779 T327 C211 P37170 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 111780 T327 C211 P37170 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 111780 T325 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 111780 T327 C211 P37170 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 111780 T327 C211 P37170 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 111781 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 111781 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 111782 T327 C211 P37170 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 111782 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 111788 T268 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 111872 T217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 111874 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49202
[junit4:junit4] 2> 111875 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 111875 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 111876 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735
[junit4:junit4] 2> 111876 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/solr.xml
[junit4:junit4] 2> 111876 T217 oasc.CoreContainer.<init> New CoreContainer 1011980385
[junit4:junit4] 2> 111877 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/'
[junit4:junit4] 2> 111877 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/'
[junit4:junit4] 2> 111924 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 111924 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 111924 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 111925 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 111925 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 111925 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 111926 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 111926 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 111926 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 111927 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 111930 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 111931 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
[junit4:junit4] 2> 111931 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 111932 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 111934 T339 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@138c115f name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 111934 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 111936 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 111940 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 111941 T341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7660cc00 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 111941 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 111945 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C212 name=collection1 org.apache.solr.core.SolrCore@13931c0f url=http://127.0.0.1:59777/_xpo/ip/collection1 node=127.0.0.1:59777__xpo%2Fip C212_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:59777__xpo%2Fip, base_url=http://127.0.0.1:59777/_xpo/ip}
[junit4:junit4] 2> 112315 T310 C212 P59777 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55519/_xpo/ip/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 112318 T310 C212 P59777 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4] 2> 112323 T310 C212 P59777 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59777/_xpo/ip START replicas=[http://127.0.0.1:55519/_xpo/ip/collection1/] nUpdates=100
[junit4:junit4] 2> 112325 T310 C212 P59777 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 112326 T310 C212 P59777 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 112326 T310 C212 P59777 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 112326 T310 C212 P59777 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 112326 T310 C212 P59777 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 112326 T310 C212 P59777 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55519/_xpo/ip/collection1/. core=collection1
[junit4:junit4] 2> 112326 T310 C212 P59777 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C213 name=collection1 org.apache.solr.core.SolrCore@1e72bd20 url=http://127.0.0.1:55519/_xpo/ip/collection1 node=127.0.0.1:55519__xpo%2Fip C213_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:55519__xpo%2Fip, base_url=http://127.0.0.1:55519/_xpo/ip, leader=true}
[junit4:junit4] 2> 112340 T253 C213 P55519 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4] 2> 112340 T255 C213 P55519 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 112343 T255 C213 P55519 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 112343 T255 C213 P55519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 112354 T255 C213 P55519 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 112355 T255 C213 P55519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 112356 T255 C213 P55519 oass.SolrIndexSearcher.<init> Opening Searcher@3c2686b8 realtime
[junit4:junit4] 2> 112356 T255 C213 P55519 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 112356 T255 C213 P55519 oasup.LogUpdateProcessor.finish [collection1] webapp=/_xpo/ip path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4] 2> 112358 T310 C212 P59777 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 112358 T310 C212 P59777 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4] 2> 112364 T310 C212 P59777 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 112383 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 112385 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37170__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37170/_xpo/ip"}
[junit4:junit4] 2> ASYNC NEW_CORE C214 name=collection1 org.apache.solr.core.SolrCore@1e72bd20 url=http://127.0.0.1:55519/_xpo/ip/collection1 node=127.0.0.1:55519__xpo%2Fip C214_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:55519__xpo%2Fip, base_url=http://127.0.0.1:55519/_xpo/ip, leader=true}
[junit4:junit4] 2> 112385 T254 C214 P55519 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 112386 T254 C214 P55519 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=20
[junit4:junit4] 2> ASYNC NEW_CORE C215 name=collection1 org.apache.solr.core.SolrCore@13931c0f url=http://127.0.0.1:59777/_xpo/ip/collection1 node=127.0.0.1:59777__xpo%2Fip C215_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:59777__xpo%2Fip, base_url=http://127.0.0.1:59777/_xpo/ip}
[junit4:junit4] 2> 112387 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 112387 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 112388 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 112391 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112391 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112392 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112391 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112391 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112392 T253 C214 P55519 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=2
[junit4:junit4] 2> 112392 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112392 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112394 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 112395 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112396 T310 C215 P59777 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344
[junit4:junit4] 2> 112397 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f36923d fullCopy=false
[junit4:junit4] 2> 112404 T255 C214 P55519 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=2
[junit4:junit4] 2> 112409 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 112415 T310 C215 P59777 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 112416 T310 C215 P59777 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 112418 T310 C215 P59777 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 112419 T310 C215 P59777 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 112419 T310 C215 P59777 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 112420 T310 C215 P59777 oass.SolrIndexSearcher.<init> Opening Searcher@4c653706 main
[junit4:junit4] 2> 112421 T309 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c653706 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 112421 T310 C215 P59777 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344;done=true>>]
[junit4:junit4] 2> 112421 T310 C215 P59777 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344
[junit4:junit4] 2> 112422 T310 C215 P59777 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344
[junit4:junit4] 2> 112422 T310 C215 P59777 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 112423 T310 C215 P59777 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 112423 T310 C215 P59777 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 112423 T310 C215 P59777 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 112426 T310 C215 P59777 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 112788 T268 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4] 2> 112789 T268 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:37170__xpo%252Fip&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4] 2> 112949 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49202__xpo%2Fip
[junit4:junit4] 2> 112950 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49202__xpo%2Fip
[junit4:junit4] 2> 112952 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112952 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4] 2> 112953 T275 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 112953 T341 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 112953 T307 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 112953 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 112954 T291 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 112954 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 112954 T259 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 112955 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 112953 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 112953 T246 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 112955 T324 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 112955 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 112955 T239 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4] 2> 112954 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 112962 T344 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 112962 T344 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 113901 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 113902 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"5",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard1",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:59777__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:59777/_xpo/ip"}
[junit4:junit4] 2> 113905 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49202__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49202/_xpo/ip"}
[junit4:junit4] 2> 113906 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 113906 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4] 2> 113910 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 113910 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 113910 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 113910 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 113910 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 113910 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 113910 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 113910 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 113985 T344 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1
[junit4:junit4] 2> 113985 T344 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 113986 T344 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 113986 T344 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 113988 T344 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1/'
[junit4:junit4] 2> 113988 T344 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1/lib/README' to classloader
[junit4:junit4] 2> 113989 T344 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 114019 T344 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 114049 T344 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 114051 T344 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 114057 T344 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 114363 T344 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 114369 T344 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 114371 T344 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 114391 T344 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 114395 T344 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 114398 T344 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 114399 T344 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 114400 T344 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 114400 T344 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 114401 T344 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 114401 T344 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 114402 T344 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 114402 T344 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/
[junit4:junit4] 2> 114402 T344 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
[junit4:junit4] 2> 114403 T344 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 114403 T344 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6
[junit4:junit4] 2> 114403 T344 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/index/
[junit4:junit4] 2> 114403 T344 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 114404 T344 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/index
[junit4:junit4] 2> 114407 T344 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 114407 T344 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 114409 T344 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 114409 T344 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 114410 T344 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 114410 T344 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 114411 T344 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 114411 T344 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 114411 T344 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 114412 T344 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 114412 T344 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 114414 T344 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 114417 T344 oass.SolrIndexSearcher.<init> Opening Searcher@693b079f main
[junit4:junit4] 2> 114418 T344 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 114418 T344 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 114422 T345 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@693b079f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 114424 T344 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 114424 T344 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49202/_xpo/ip collection:collection1 shard:shard3
[junit4:junit4] 2> 114428 T344 oasc.ZkController.register We are http://127.0.0.1:49202/_xpo/ip/collection1/ and leader is http://127.0.0.1:44496/_xpo/ip/collection1/
[junit4:junit4] 2> 114428 T344 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49202/_xpo/ip
[junit4:junit4] 2> 114428 T344 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 114429 T344 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C216 name=collection1 org.apache.solr.core.SolrCore@29709fba url=http://127.0.0.1:49202/_xpo/ip/collection1 node=127.0.0.1:49202__xpo%2Fip C216_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49202__xpo%2Fip, base_url=http://127.0.0.1:49202/_xpo/ip}
[junit4:junit4] 2> 114429 T346 C216 P49202 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 114430 T346 C216 P49202 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 114430 T344 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 114430 T346 C216 P49202 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4] 2> 114430 T346 C216 P49202 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 114431 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
[junit4:junit4] 2> 114432 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4] 2> 114432 T346 C216 P49202 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 114432 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> 114440 T286 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4] 2> 114499 T217 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4] 2> 114501 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47253
[junit4:junit4] 2> 114502 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4] 2> 114502 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4] 2> 114502 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386
[junit4:junit4] 2> 114502 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/solr.xml
[junit4:junit4] 2> 114503 T217 oasc.CoreContainer.<init> New CoreContainer 428115399
[junit4:junit4] 2> 114503 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/'
[junit4:junit4] 2> 114503 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/'
[junit4:junit4] 2> 114543 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4] 2> 114543 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4] 2> 114543 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4] 2> 114544 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4] 2> 114544 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4] 2> 114544 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4] 2> 114545 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4] 2> 114545 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4] 2> 114545 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4] 2> 114546 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4] 2> 114549 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4] 2> 114549 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
[junit4:junit4] 2> 114550 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4] 2> 114551 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 114552 T358 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@179fc2d2 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 114553 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 114556 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4] 2> 114559 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4] 2> 114561 T360 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dac7392 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4] 2> 114561 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4] 2> 114568 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4] 2> ASYNC NEW_CORE C217 name=collection1 org.apache.solr.core.SolrCore@a0a3409 url=http://127.0.0.1:37170/_xpo/ip/collection1 node=127.0.0.1:37170__xpo%2Fip C217_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:37170__xpo%2Fip, base_url=http://127.0.0.1:37170/_xpo/ip}
[junit4:junit4] 2> 114790 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:58042/_xpo/ip/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4] 2> 114790 T327 C217 P37170 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37170/_xpo/ip START replicas=[http://127.0.0.1:58042/_xpo/ip/collection1/] nUpdates=100
[junit4:junit4] 2> 114791 T327 C217 P37170 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4] 2> 114791 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4] 2> 114791 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4] 2> 114791 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4] 2> 114791 T327 C217 P37170 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4] 2> 114792 T327 C217 P37170 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:58042/_xpo/ip/collection1/. core=collection1
[junit4:junit4] 2> 114792 T327 C217 P37170 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4] 2> ASYNC NEW_CORE C218 name=collection1 org.apache.solr.core.SolrCore@7633a644 url=http://127.0.0.1:58042/_xpo/ip/collection1 node=127.0.0.1:58042__xpo%2Fip C218_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:58042__xpo%2Fip, base_url=http://127.0.0.1:58042/_xpo/ip, leader=true}
[junit4:junit4] 2> 114800 T268 C218 P58042 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4] 2> 114800 T271 C218 P58042 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4] 2> 114805 T271 C218 P58042 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 114806 T271 C218 P58042 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 114815 T271 C218 P58042 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 114815 T271 C218 P58042 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 114815 T271 C218 P58042 oass.SolrIndexSearcher.<init> Opening Searcher@593c82ea realtime
[junit4:junit4] 2> 114816 T271 C218 P58042 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4] 2> 114816 T271 C218 P58042 oasup.LogUpdateProcessor.finish [collection1] webapp=/_xpo/ip path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4] 2> 114817 T327 C217 P37170 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 114817 T327 C217 P37170 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4] 2> 114820 T268 C218 P58042 oash.ReplicationHandler.inform Commits will be reserved for 10000
[junit4:junit4] 2> 114820 T268 C218 P58042 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4] 2> 114821 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4] 2> 114822 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4] 2> 114822 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4] 2> 114824 T271 C218 P58042 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4] 2> 114825 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4] 2> 114826 T327 C217 P37170 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774
[junit4:junit4] 2> 114826 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7cf2a936 fullCopy=false
[junit4:junit4] 2> 114829 T270 C218 P58042 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4] 2> 114830 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4] 2> 114835 T327 C217 P37170 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4] 2> 114835 T327 C217 P37170 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4] 2> 114837 T327 C217 P37170 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4] 2> 114837 T327 C217 P37170 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4] 2> 114838 T327 C217 P37170 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4] 2> 114838 T327 C217 P37170 oass.SolrIndexSearcher.<init> Opening Searcher@550371aa main
[junit4:junit4] 2> 114839 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@550371aa main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4] 2> 114839 T327 C217 P37170 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774;done=true>>]
[junit4:junit4] 2> 114839 T327 C217 P37170 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774
[junit4:junit4] 2> 114840 T327 C217 P37170 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774
[junit4:junit4] 2> 114840 T327 C217 P37170 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4] 2> 114840 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4] 2> 114840 T327 C217 P37170 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4] 2> 114840 T327 C217 P37170 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 114842 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4] 2> 115419 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 115420 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"7",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard3",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"recovering",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:49202__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:49202/_xpo/ip"}
[junit4:junit4] 2> 115423 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"6",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"active",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:37170__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:37170/_xpo/ip"}
[junit4:junit4] 2> 115445 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115445 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115445 T360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115445 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115445 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115445 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115445 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115445 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115445 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115573 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47253__xpo%2Fip
[junit4:junit4] 2> 115574 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47253__xpo%2Fip
[junit4:junit4] 2> 115576 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115576 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4] 2> 115577 T341 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115577 T275 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115577 T259 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115578 T307 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115578 T246 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115577 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 115578 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 115578 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 115578 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 115578 T291 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115578 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 115579 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 115581 T360 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115581 T360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 115581 T239 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115582 T324 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4] 2> 115589 T363 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 115589 T363 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 116441 T286 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
[junit4:junit4] 2> 116441 T286 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:49202__xpo%252Fip&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001
[junit4:junit4] 2> 116955 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 116956 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"8",
[junit4:junit4] 2> "numShards":"3",
[junit4:junit4] 2> "shard":null,
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:47253__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:47253/_xpo/ip"}
[junit4:junit4] 2> 116956 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4] 2> 116957 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4] 2> 116963 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 116963 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 116963 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 116963 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 116963 T360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 116963 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 116963 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 116963 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 116963 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4] 2> 117590 T363 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1
[junit4:junit4] 2> 117591 T363 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4] 2> 117592 T363 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4] 2> 117592 T363 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4] 2> 117593 T363 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1/'
[junit4:junit4] 2> 117594 T363 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1/lib/README' to classloader
[junit4:junit4] 2> 117594 T363 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1/lib/classes/' to classloader
[junit4:junit4] 2> 117624 T363 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
[junit4:junit4] 2> 117658 T363 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4] 2> 117660 T363 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4] 2> 117668 T363 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4] 2> 118003 T363 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4] 2> 118009 T363 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4] 2> 118011 T363 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4] 2> 118036 T363 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 118041 T363 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4] 2> 118046 T363 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 118048 T363 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 118048 T363 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 118049 T363 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4] 2> 118050 T363 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4] 2> 118050 T363 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4] 2> 118051 T363 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 118051 T363 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/
[junit4:junit4] 2> 118051 T363 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
[junit4:junit4] 2> 118052 T363 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4] 2> 118053 T363 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7
[junit4:junit4] 2> 118053 T363 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/index/
[junit4:junit4] 2> 118054 T363 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4] 2> 118054 T363 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/index
[junit4:junit4] 2> 118058 T363 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4] 2> 118058 T363 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4] 2> 118061 T363 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4] 2> 118061 T363 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4] 2> 118062 T363 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4] 2> 118063 T363 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4] 2> 118063 T363 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4] 2> 118063 T363 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4] 2> 118064 T363 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4] 2> 118064 T363 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4] 2> 118065 T363 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4] 2> 118069 T363 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4] 2> 118073 T363 oass.SolrIndexSearcher.<init> Opening Searcher@57fef174 main
[junit4:junit4] 2> 118074 T363 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4] 2> 118074 T363 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4] 2> 118079 T364 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@57fef174 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4] 2> 118080 T363 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4] 2> 118081 T363 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47253/_xpo/ip collection:collection1 shard:shard1
[junit4:junit4] 2> 118085 T363 oasc.ZkController.register We are http://127.0.0.1:47253/_xpo/ip/collection1/ and leader is http://127.0.0.1:55519/_xpo/ip/collection1/
[junit4:junit4] 2> 118085 T363 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47253/_xpo/ip
[junit4:junit4] 2> 118085 T363 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4] 2> 118085 T363 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4] 2> ASYNC NEW_CORE C219 name=collection1 org.apache.solr.core.SolrCore@2cfa3e0e url=http://127.0.0.1:47253/_xpo/ip/collection1 node=127.0.0.1:47253__xpo%2Fip C219_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47253__xpo%2Fip, base_url=http://127.0.0.1:47253/_xpo/ip}
[junit4:junit4] 2> 118086 T365 C219 P47253 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4:junit4] 2> 118086 T365 C219 P47253 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4] 2> 118086 T363
[...truncated too long message...]
the main queue
[junit4:junit4] 2> 346295 T715 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4] 2> 346296 T715 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"12",
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "node_name":"127.0.0.1:40497__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40497/_xpo/ip"}
[junit4:junit4] 2> 346299 T715 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4] 2> "operation":"state",
[junit4:junit4] 2> "core_node_name":"12",
[junit4:junit4] 2> "numShards":null,
[junit4:junit4] 2> "shard":"shard2",
[junit4:junit4] 2> "roles":null,
[junit4:junit4] 2> "state":"down",
[junit4:junit4] 2> "shard_state":"active",
[junit4:junit4] 2> "core":"collection1",
[junit4:junit4] 2> "collection":"collection1",
[junit4:junit4] 2> "shard_range":null,
[junit4:junit4] 2> "node_name":"127.0.0.1:40497__xpo%2Fip",
[junit4:junit4] 2> "base_url":"http://127.0.0.1:40497/_xpo/ip"}
[junit4:junit4] 2> 346305 T457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 346307 T217 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_xpo/ip,null}
[junit4:junit4] 2> 346359 T217 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 37443
[junit4:junit4] 2> 346360 T217 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=695679923
[junit4:junit4] 2> 346498 T419 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 346498 T419 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4] 2> 346498 T419 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 347361 T217 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4] 2> 347362 T217 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4] 2> 347364 T217 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@287ab2c
[junit4:junit4] 2> 347376 T217 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=22,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1242,cumulative_deletesById=596,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4] 2> 347376 T217 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4] 2> 347377 T217 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4] 2> 347377 T217 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4] 2> 347378 T217 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4] 2> 347379 T217 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4] 2> 347379 T217 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12;done=false>>]
[junit4:junit4] 2> 347379 T217 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12
[junit4:junit4] 2> 347379 T217 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12/index;done=false>>]
[junit4:junit4] 2> 347380 T217 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12/index
[junit4:junit4] 2> 347381 T715 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89881105220763676-127.0.0.1:37443__xpo%2Fip-n_0000000012) am no longer a leader.
[junit4:junit4] 2> 347404 T217 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_xpo/ip,null}
[junit4:junit4] 2> 347473 T217 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4] 2> 347475 T217 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35312 35312
[junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4] 1> safe stop:
[junit4:junit4] 1> safe stop:
[junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=62718A172099C606 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_CS -Dtests.timezone=Antarctica/Davis -Dtests.file.encoding=ISO-8859-1
[junit4:junit4] FAILURE 254s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4] > Throwable #1: java.lang.AssertionError: shard3 is not consistent. Got 648 from http://127.0.0.1:49202/_xpo/ip/collection1lastClient and got 646 from http://127.0.0.1:37443/_xpo/ip/collection1
[junit4:junit4] > at __randomizedtesting.SeedInfo.seed([62718A172099C606:E397040F57C6A63A]:0)
[junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
[junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] 2> 347520 T217 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4] 2> 254309 T216 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4] 2> 347585 T436 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 347586 T436 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4] 2> 347586 T436 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 348682 T457 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> 348682 T457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4] 2> 348682 T457 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=324), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=324)), sim=DefaultSimilarity, locale=sr_CS, timezone=Antarctica/Davis
[junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=4,free=89012592,total=317915136
[junit4:junit4] 2> NOTE: All tests run in this JVM: [TestUpdate, BadIndexSchemaTest, IndexBasedSpellCheckerTest, TestQuerySenderNoQuery, NumericFieldsTest, TestRandomFaceting, TestFieldTypeResource, HighlighterConfigTest, TestAddFieldRealTimeGet, TestSolrQueryParserDefaultOperatorResource, DirectSolrSpellCheckerTest, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 255.60s, 1 test, 1 failure <<< FAILURES!
[...truncated 816 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 298 suites, 1252 tests, 1 failure, 14 ignored (8 assumptions)
Total time: 39 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
Re: [JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_21) - Build # 6105 - Still Failing!
Posted by Mark Miller <ma...@gmail.com>.
Hopefully I will be able to spend some time and get to the bottom of these soon.
Oddly enough, I think it was the change to how compound file format was set that is causing this. Which is strange and interesting.
- Mark
On Jun 17, 2013, at 10:06 AM, Policeman Jenkins Server <je...@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6105/
> Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
>
> 1 tests failed.
> REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
>
> Error Message:
> shard3 is not consistent. Got 648 from http://127.0.0.1:49202/_xpo/ip/collection1lastClient and got 646 from http://127.0.0.1:37443/_xpo/ip/collection1
>
> Stack Trace:
> java.lang.AssertionError: shard3 is not consistent. Got 648 from http://127.0.0.1:49202/_xpo/ip/collection1lastClient and got 646 from http://127.0.0.1:37443/_xpo/ip/collection1
> at __randomizedtesting.SeedInfo.seed([62718A172099C606:E397040F57C6A63A]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
> at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
> at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
> at java.lang.Thread.run(Thread.java:722)
>
>
>
>
> Build Log:
> [...truncated 9329 lines...]
> [junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
> [junit4:junit4] 2> 93216 T217 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_xpo/ip
> [junit4:junit4] 2> 93233 T217 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
> [junit4:junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371476825183
> [junit4:junit4] 2> 93238 T217 oasc.ZkTestServer.run STARTING ZK TEST SERVER
> [junit4:junit4] 2> 93241 T218 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
> [junit4:junit4] 2> 93341 T217 oasc.ZkTestServer.run start zk server on port:35312
> [junit4:junit4] 2> 93386 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 93537 T224 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3756a51e name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 93537 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 93539 T217 oascc.SolrZkClient.makePath makePath: /solr
> [junit4:junit4] 2> 93576 T219 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13f526394b40000, likely client has closed socket
> [junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220)
> [junit4:junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> [junit4:junit4] 2> at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4] 2>
> [junit4:junit4] 2> 93580 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 93582 T226 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@636c241c name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 93583 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 93585 T217 oascc.SolrZkClient.makePath makePath: /collections/collection1
> [junit4:junit4] 2> 93591 T217 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
> [junit4:junit4] 2> 93599 T217 oascc.SolrZkClient.makePath makePath: /collections/control_collection
> [junit4:junit4] 2> 93602 T217 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
> [junit4:junit4] 2> 93607 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
> [junit4:junit4] 2> 93608 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
> [junit4:junit4] 2> 93615 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
> [junit4:junit4] 2> 93615 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
> [junit4:junit4] 2> 93620 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
> [junit4:junit4] 2> 93621 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
> [junit4:junit4] 2> 93626 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
> [junit4:junit4] 2> 93626 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
> [junit4:junit4] 2> 93630 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
> [junit4:junit4] 2> 93631 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
> [junit4:junit4] 2> 93644 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
> [junit4:junit4] 2> 93645 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
> [junit4:junit4] 2> 93649 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4:junit4] 2> 93650 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
> [junit4:junit4] 2> 93654 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
> [junit4:junit4] 2> 93655 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
> [junit4:junit4] 2> 93659 T217 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
> [junit4:junit4] 2> 93660 T217 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
> [junit4:junit4] 2> 93751 T217 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 93758 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53862
> [junit4:junit4] 2> 93759 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 93759 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 93759 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617
> [junit4:junit4] 2> 93760 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/solr.xml
> [junit4:junit4] 2> 93760 T217 oasc.CoreContainer.<init> New CoreContainer 1639190412
> [junit4:junit4] 2> 93760 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/'
> [junit4:junit4] 2> 93761 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/'
> [junit4:junit4] 2> 93806 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 93807 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 93807 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 93807 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 93808 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 93808 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 93808 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 93809 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 93809 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 93809 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 93862 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 93862 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
> [junit4:junit4] 2> 93868 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 93869 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 93871 T237 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19e0ddff name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 93872 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 93876 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 93882 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 93883 T239 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a4360d5 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 93884 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 93889 T217 oascc.SolrZkClient.makePath makePath: /overseer/queue
> [junit4:junit4] 2> 93897 T217 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
> [junit4:junit4] 2> 93905 T217 oascc.SolrZkClient.makePath makePath: /live_nodes
> [junit4:junit4] 2> 93908 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53862__xpo%2Fip
> [junit4:junit4] 2> 93911 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53862__xpo%2Fip
> [junit4:junit4] 2> 93916 T217 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
> [junit4:junit4] 2> 93928 T217 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
> [junit4:junit4] 2> 93932 T217 oasc.Overseer.start Overseer (id=89881105220763651-127.0.0.1:53862__xpo%2Fip-n_0000000000) starting
> [junit4:junit4] 2> 93948 T217 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
> [junit4:junit4] 2> 93963 T241 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
> [junit4:junit4] 2> 93963 T217 oascc.SolrZkClient.makePath makePath: /clusterstate.json
> [junit4:junit4] 2> 93968 T217 oascc.SolrZkClient.makePath makePath: /aliases.json
> [junit4:junit4] 2> 93972 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 93984 T240 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
> [junit4:junit4] 2> 93992 T242 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 93993 T242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 95491 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 95493 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"1",
> [junit4:junit4] 2> "numShards":"1",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"control_collection",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:53862__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:53862/_xpo/ip"}
> [junit4:junit4] 2> 95493 T240 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
> [junit4:junit4] 2> 95504 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 95520 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 95995 T242 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1
> [junit4:junit4] 2> 95996 T242 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
> [junit4:junit4] 2> 95997 T242 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 95997 T242 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
> [junit4:junit4] 2> 96000 T242 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1/'
> [junit4:junit4] 2> 96001 T242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1/lib/README' to classloader
> [junit4:junit4] 2> 96001 T242 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 96049 T242 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 96091 T242 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 96094 T242 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 96115 T242 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 96490 T242 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 96498 T242 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 96501 T242 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 96528 T242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 96533 T242 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 96538 T242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 96540 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 96540 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 96540 T242 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 96542 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 96542 T242 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 96542 T242 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 96549 T242 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371476825617/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/
> [junit4:junit4] 2> 96549 T242 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
> [junit4:junit4] 2> 96550 T242 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 96550 T242 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data
> [junit4:junit4] 2> 96551 T242 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/index/
> [junit4:junit4] 2> 96551 T242 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 96551 T242 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/index
> [junit4:junit4] 2> 96555 T242 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/control/data/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 96555 T242 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 96557 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 96558 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 96559 T242 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 96559 T242 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 96560 T242 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 96560 T242 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 96560 T242 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 96560 T242 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 96561 T242 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 96563 T242 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 96566 T242 oass.SolrIndexSearcher.<init> Opening Searcher@69665ade main
> [junit4:junit4] 2> 96567 T242 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 96567 T242 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 96571 T243 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@69665ade main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 96573 T242 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 96573 T242 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53862/_xpo/ip collection:control_collection shard:shard1
> [junit4:junit4] 2> 96578 T242 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
> [junit4:junit4] 2> 96586 T242 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
> [junit4:junit4] 2> 96599 T242 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 96603 T242 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 96604 T242 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 96605 T242 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53862/_xpo/ip/collection1/
> [junit4:junit4] 2> 96606 T242 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 96606 T242 oasc.SyncStrategy.syncToMe http://127.0.0.1:53862/_xpo/ip/collection1/ has no replicas
> [junit4:junit4] 2> 96607 T242 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53862/_xpo/ip/collection1/
> [junit4:junit4] 2> 96607 T242 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
> [junit4:junit4] 2> 97028 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 97039 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 97069 T242 oasc.ZkController.register We are http://127.0.0.1:53862/_xpo/ip/collection1/ and leader is http://127.0.0.1:53862/_xpo/ip/collection1/
> [junit4:junit4] 2> 97069 T242 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53862/_xpo/ip
> [junit4:junit4] 2> 97069 T242 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 97069 T242 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 97070 T242 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 97072 T242 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 97074 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 97074 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 97088 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 97094 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:
> [junit4:junit4] 2> 97096 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 97098 T246 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cd647e3 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 97098 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 97100 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 97104 T217 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
> [junit4:junit4] 2> 97168 T217 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 97170 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55519
> [junit4:junit4] 2> 97170 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 97171 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 97171 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053
> [junit4:junit4] 2> 97171 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/solr.xml
> [junit4:junit4] 2> 97171 T217 oasc.CoreContainer.<init> New CoreContainer 849998145
> [junit4:junit4] 2> 97172 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/'
> [junit4:junit4] 2> 97172 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/'
> [junit4:junit4] 2> 97226 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 97227 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 97227 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 97227 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 97228 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 97228 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 97228 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 97229 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 97229 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 97230 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 97236 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 97237 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
> [junit4:junit4] 2> 97237 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 97238 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 97240 T257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@478bb673 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 97241 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 97243 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 97250 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 97252 T259 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@406993a1 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 97252 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 97258 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 98262 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55519__xpo%2Fip
> [junit4:junit4] 2> 98264 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55519__xpo%2Fip
> [junit4:junit4] 2> 98266 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 98268 T246 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 98268 T259 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 98269 T239 oascc.ZkStateReader$3.process Updating live nodes... (2)
> [junit4:junit4] 2> 98276 T260 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 98276 T260 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 98546 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 98547 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"1",
> [junit4:junit4] 2> "numShards":"1",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"control_collection",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:53862__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:53862/_xpo/ip"}
> [junit4:junit4] 2> 98552 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"2",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:55519__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:55519/_xpo/ip"}
> [junit4:junit4] 2> 98552 T240 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
> [junit4:junit4] 2> 98552 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 98558 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 98558 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 98558 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 99278 T260 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1
> [junit4:junit4] 2> 99278 T260 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 99279 T260 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 99280 T260 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 99282 T260 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1/'
> [junit4:junit4] 2> 99282 T260 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1/lib/README' to classloader
> [junit4:junit4] 2> 99283 T260 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 99310 T260 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 99360 T260 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 99363 T260 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 99385 T260 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 99717 T260 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 99728 T260 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 99731 T260 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 99761 T260 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 99766 T260 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 99771 T260 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 99773 T260 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 99774 T260 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 99774 T260 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 99776 T260 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 99776 T260 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 99776 T260 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 99777 T260 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371476829053/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/
> [junit4:junit4] 2> 99777 T260 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
> [junit4:junit4] 2> 99777 T260 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 99778 T260 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1
> [junit4:junit4] 2> 99778 T260 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index/
> [junit4:junit4] 2> 99779 T260 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 99779 T260 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index
> [junit4:junit4] 2> 99782 T260 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 99782 T260 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 99784 T260 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 99785 T260 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 99785 T260 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 99786 T260 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 99786 T260 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 99787 T260 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 99787 T260 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 99787 T260 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 99788 T260 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 99790 T260 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 99795 T260 oass.SolrIndexSearcher.<init> Opening Searcher@254c25e1 main
> [junit4:junit4] 2> 99796 T260 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 99796 T260 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 99802 T260 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 99802 T260 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55519/_xpo/ip collection:collection1 shard:shard1
> [junit4:junit4] 2> 99803 T260 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
> [junit4:junit4] 2> 99804 T261 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@254c25e1 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 99812 T260 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 99817 T260 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 99818 T260 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 99818 T260 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55519/_xpo/ip/collection1/
> [junit4:junit4] 2> 99818 T260 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 99818 T260 oasc.SyncStrategy.syncToMe http://127.0.0.1:55519/_xpo/ip/collection1/ has no replicas
> [junit4:junit4] 2> 99819 T260 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55519/_xpo/ip/collection1/
> [junit4:junit4] 2> 99819 T260 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
> [junit4:junit4] 2> 100067 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 100155 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 100155 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 100155 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 100177 T260 oasc.ZkController.register We are http://127.0.0.1:55519/_xpo/ip/collection1/ and leader is http://127.0.0.1:55519/_xpo/ip/collection1/
> [junit4:junit4] 2> 100178 T260 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55519/_xpo/ip
> [junit4:junit4] 2> 100178 T260 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 100178 T260 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 100178 T260 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 100181 T260 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 100182 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 100183 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 100183 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 100250 T217 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 100252 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58042
> [junit4:junit4] 2> 100253 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 100254 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 100254 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136
> [junit4:junit4] 2> 100254 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/solr.xml
> [junit4:junit4] 2> 100254 T217 oasc.CoreContainer.<init> New CoreContainer 1776184614
> [junit4:junit4] 2> 100255 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/'
> [junit4:junit4] 2> 100255 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/'
> [junit4:junit4] 2> 100319 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 100320 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 100320 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 100321 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 100321 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 100322 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 100322 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 100323 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 100323 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 100323 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 100329 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 100330 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
> [junit4:junit4] 2> 100331 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 100332 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 100334 T273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66161e13 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 100334 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 100338 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 100343 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 100347 T275 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bc8ef7b name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 100347 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 100360 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 101364 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58042__xpo%2Fip
> [junit4:junit4] 2> 101366 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58042__xpo%2Fip
> [junit4:junit4] 2> 101368 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 101369 T246 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 101370 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 101372 T259 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 101372 T239 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 101372 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 101372 T275 oascc.ZkStateReader$3.process Updating live nodes... (3)
> [junit4:junit4] 2> 101381 T276 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 101381 T276 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 101666 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 101667 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"2",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:55519__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:55519/_xpo/ip"}
> [junit4:junit4] 2> 101671 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"3",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:58042__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:58042/_xpo/ip"}
> [junit4:junit4] 2> 101672 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 101672 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4] 2> 101679 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 101679 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 101679 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 101679 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 102383 T276 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1
> [junit4:junit4] 2> 102383 T276 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 102384 T276 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 102385 T276 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 102387 T276 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1/'
> [junit4:junit4] 2> 102388 T276 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1/lib/README' to classloader
> [junit4:junit4] 2> 102388 T276 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 102426 T276 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 102464 T276 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 102466 T276 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 102482 T276 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 102878 T276 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 102886 T276 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 102888 T276 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 102913 T276 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 102917 T276 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 102923 T276 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 102925 T276 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 102925 T276 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 102928 T276 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 102930 T276 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 102930 T276 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 102930 T276 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 102931 T276 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371476832136/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/
> [junit4:junit4] 2> 102931 T276 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
> [junit4:junit4] 2> 102931 T276 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 102932 T276 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2
> [junit4:junit4] 2> 102932 T276 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index/
> [junit4:junit4] 2> 102932 T276 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 102933 T276 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index
> [junit4:junit4] 2> 102936 T276 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 102937 T276 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 102940 T276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 102940 T276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 102941 T276 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 102941 T276 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 102942 T276 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 102942 T276 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 102943 T276 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 102943 T276 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 102944 T276 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 102946 T276 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 102949 T276 oass.SolrIndexSearcher.<init> Opening Searcher@7e74b92b main
> [junit4:junit4] 2> 102950 T276 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 102950 T276 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 102954 T277 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e74b92b main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 102956 T276 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 102957 T276 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58042/_xpo/ip collection:collection1 shard:shard2
> [junit4:junit4] 2> 102958 T276 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
> [junit4:junit4] 2> 102967 T276 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 102972 T276 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 102972 T276 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 102972 T276 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58042/_xpo/ip/collection1/
> [junit4:junit4] 2> 102973 T276 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 102973 T276 oasc.SyncStrategy.syncToMe http://127.0.0.1:58042/_xpo/ip/collection1/ has no replicas
> [junit4:junit4] 2> 102973 T276 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58042/_xpo/ip/collection1/
> [junit4:junit4] 2> 102973 T276 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
> [junit4:junit4] 2> 103192 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 103203 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 103203 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 103203 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 103203 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 103230 T276 oasc.ZkController.register We are http://127.0.0.1:58042/_xpo/ip/collection1/ and leader is http://127.0.0.1:58042/_xpo/ip/collection1/
> [junit4:junit4] 2> 103231 T276 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58042/_xpo/ip
> [junit4:junit4] 2> 103231 T276 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 103231 T276 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 103231 T276 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 103234 T276 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 103236 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 103236 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 103237 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 103337 T217 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 103340 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44496
> [junit4:junit4] 2> 103341 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 103341 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 103342 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191
> [junit4:junit4] 2> 103343 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/solr.xml
> [junit4:junit4] 2> 103343 T217 oasc.CoreContainer.<init> New CoreContainer 248526691
> [junit4:junit4] 2> 103344 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/'
> [junit4:junit4] 2> 103344 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/'
> [junit4:junit4] 2> 103411 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 103411 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 103411 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 103412 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 103412 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 103412 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 103413 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 103413 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 103413 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 103414 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 103418 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 103418 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
> [junit4:junit4] 2> 103419 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 103420 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 103422 T289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e3aae18 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 103422 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 103426 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 103432 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 103436 T291 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f8cd9e1 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 103437 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 103443 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 104448 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44496__xpo%2Fip
> [junit4:junit4] 2> 104449 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44496__xpo%2Fip
> [junit4:junit4] 2> 104452 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 104454 T246 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 104454 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 104454 T275 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 104454 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 104454 T259 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 104455 T291 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 104455 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 104456 T239 oascc.ZkStateReader$3.process Updating live nodes... (4)
> [junit4:junit4] 2> 104463 T292 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 104463 T292 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 104712 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 104713 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"3",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:58042__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:58042/_xpo/ip"}
> [junit4:junit4] 2> 104718 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"4",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:44496__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:44496/_xpo/ip"}
> [junit4:junit4] 2> 104718 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 104718 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
> [junit4:junit4] 2> 104724 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 104724 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 104725 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 104724 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 104724 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 105465 T292 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1
> [junit4:junit4] 2> 105465 T292 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 105467 T292 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 105468 T292 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 105471 T292 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1/'
> [junit4:junit4] 2> 105472 T292 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1/lib/README' to classloader
> [junit4:junit4] 2> 105473 T292 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 105509 T292 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 105546 T292 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 105547 T292 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 105556 T292 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 105925 T292 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 105934 T292 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 105938 T292 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 105964 T292 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 105968 T292 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 105972 T292 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 105974 T292 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 105975 T292 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 105975 T292 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 105976 T292 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 105976 T292 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 105977 T292 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 105977 T292 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371476835191/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/
> [junit4:junit4] 2> 105977 T292 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
> [junit4:junit4] 2> 105978 T292 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 105978 T292 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3
> [junit4:junit4] 2> 105978 T292 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/index/
> [junit4:junit4] 2> 105979 T292 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 105979 T292 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/index
> [junit4:junit4] 2> 105983 T292 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty3/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 105983 T292 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 105986 T292 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 105987 T292 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 105987 T292 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 105988 T292 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 105989 T292 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 105989 T292 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 105989 T292 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 105990 T292 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 105991 T292 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 105995 T292 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 105999 T292 oass.SolrIndexSearcher.<init> Opening Searcher@7e9dbe8a main
> [junit4:junit4] 2> 106000 T292 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 106000 T292 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 106006 T293 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e9dbe8a main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 106008 T292 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 106008 T292 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44496/_xpo/ip collection:collection1 shard:shard3
> [junit4:junit4] 2> 106009 T292 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
> [junit4:junit4] 2> 106017 T292 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
> [junit4:junit4] 2> 106021 T292 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
> [junit4:junit4] 2> 106021 T292 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
> [junit4:junit4] 2> 106021 T292 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44496/_xpo/ip/collection1/
> [junit4:junit4] 2> 106021 T292 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
> [junit4:junit4] 2> 106022 T292 oasc.SyncStrategy.syncToMe http://127.0.0.1:44496/_xpo/ip/collection1/ has no replicas
> [junit4:junit4] 2> 106022 T292 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44496/_xpo/ip/collection1/
> [junit4:junit4] 2> 106022 T292 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
> [junit4:junit4] 2> 106238 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 106248 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 106248 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 106248 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 106249 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 106252 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 106280 T292 oasc.ZkController.register We are http://127.0.0.1:44496/_xpo/ip/collection1/ and leader is http://127.0.0.1:44496/_xpo/ip/collection1/
> [junit4:junit4] 2> 106280 T292 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44496/_xpo/ip
> [junit4:junit4] 2> 106280 T292 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
> [junit4:junit4] 2> 106281 T292 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 106281 T292 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 106284 T292 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 106285 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 106286 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 106286 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 106375 T217 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 106378 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59777
> [junit4:junit4] 2> 106379 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 106380 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 106380 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239
> [junit4:junit4] 2> 106380 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/solr.xml
> [junit4:junit4] 2> 106381 T217 oasc.CoreContainer.<init> New CoreContainer 1160373085
> [junit4:junit4] 2> 106381 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/'
> [junit4:junit4] 2> 106381 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/'
> [junit4:junit4] 2> 106439 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 106440 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 106440 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 106441 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 106441 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 106441 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 106442 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 106442 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 106442 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 106443 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 106446 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 106447 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
> [junit4:junit4] 2> 106447 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 106448 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 106450 T305 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@136cf name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 106451 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 106453 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 106457 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 106459 T307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6415cf5a name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 106459 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 106464 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 107468 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59777__xpo%2Fip
> [junit4:junit4] 2> 107470 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59777__xpo%2Fip
> [junit4:junit4] 2> 107473 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
> [junit4:junit4] 2> 107475 T307 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 107475 T259 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 107476 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107476 T291 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 107477 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107476 T246 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 107476 T275 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 107478 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107478 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107479 T239 oascc.ZkStateReader$3.process Updating live nodes... (5)
> [junit4:junit4] 2> 107486 T308 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 107486 T308 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 107761 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 107762 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"4",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard3",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:44496__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:44496/_xpo/ip"}
> [junit4:junit4] 2> 107767 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"5",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:59777__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:59777/_xpo/ip"}
> [junit4:junit4] 2> 107768 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 107768 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 107835 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107835 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107835 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107835 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107835 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 107835 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 108488 T308 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1
> [junit4:junit4] 2> 108489 T308 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 108490 T308 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 108490 T308 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 108491 T308 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1/'
> [junit4:junit4] 2> 108492 T308 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1/lib/README' to classloader
> [junit4:junit4] 2> 108493 T308 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 108532 T308 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 108569 T308 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 108571 T308 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 108578 T308 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 108907 T308 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 108913 T308 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 108916 T308 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 108937 T308 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 108942 T308 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 108949 T308 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 108951 T308 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 108951 T308 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 108951 T308 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 108953 T308 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 108953 T308 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 108954 T308 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 108954 T308 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371476838239/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/
> [junit4:junit4] 2> 108954 T308 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
> [junit4:junit4] 2> 108955 T308 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 108955 T308 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4
> [junit4:junit4] 2> 108955 T308 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index/
> [junit4:junit4] 2> 108956 T308 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 108956 T308 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index
> [junit4:junit4] 2> 109026 T308 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 109026 T308 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 109029 T308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 109030 T308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 109030 T308 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 109031 T308 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 109032 T308 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 109032 T308 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 109033 T308 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 109033 T308 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 109034 T308 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 109037 T308 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 109041 T308 oass.SolrIndexSearcher.<init> Opening Searcher@70c8ea27 main
> [junit4:junit4] 2> 109042 T308 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 109043 T308 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 109047 T309 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@70c8ea27 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 109050 T308 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 109050 T308 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59777/_xpo/ip collection:collection1 shard:shard1
> [junit4:junit4] 2> 109054 T308 oasc.ZkController.register We are http://127.0.0.1:59777/_xpo/ip/collection1/ and leader is http://127.0.0.1:55519/_xpo/ip/collection1/
> [junit4:junit4] 2> 109054 T308 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59777/_xpo/ip
> [junit4:junit4] 2> 109054 T308 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 109054 T308 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C210 name=collection1 org.apache.solr.core.SolrCore@13931c0f url=http://127.0.0.1:59777/_xpo/ip/collection1 node=127.0.0.1:59777__xpo%2Fip C210_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:59777__xpo%2Fip, base_url=http://127.0.0.1:59777/_xpo/ip}
> [junit4:junit4] 2> 109058 T310 C210 P59777 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 109059 T310 C210 P59777 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 109059 T308 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 109059 T310 C210 P59777 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 109059 T310 C210 P59777 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 109060 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 109060 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 109061 T310 C210 P59777 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 109061 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 109164 T217 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 109166 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37170
> [junit4:junit4] 2> 109167 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 109167 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 109168 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015
> [junit4:junit4] 2> 109168 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/solr.xml
> [junit4:junit4] 2> 109169 T217 oasc.CoreContainer.<init> New CoreContainer 175101158
> [junit4:junit4] 2> 109169 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/'
> [junit4:junit4] 2> 109170 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/'
> [junit4:junit4] 2> 109222 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 109222 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 109223 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 109223 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 109223 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 109224 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 109224 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 109224 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 109225 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 109225 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 109229 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 109229 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
> [junit4:junit4] 2> 109230 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 109230 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 109232 T322 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@75e9584 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 109232 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 109234 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 109238 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 109239 T324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@19b07bac name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 109239 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 109244 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> 109284 T254 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 109351 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 109353 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"5",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:59777__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:59777/_xpo/ip"}
> [junit4:junit4] 2> 109359 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 109360 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 109360 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 109360 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 109360 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 109359 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 109359 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 110248 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37170__xpo%2Fip
> [junit4:junit4] 2> 110249 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37170__xpo%2Fip
> [junit4:junit4] 2> 110253 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 110253 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
> [junit4:junit4] 2> 110254 T307 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 110254 T291 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 110255 T259 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 110254 T275 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 110254 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110254 T246 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 110255 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110255 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110255 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110257 T324 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 110257 T239 oascc.ZkStateReader$3.process Updating live nodes... (6)
> [junit4:junit4] 2> 110256 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110264 T325 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 110264 T325 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 110286 T254 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4] 2> 110286 T254 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=5&state=recovering&nodeName=127.0.0.1:59777__xpo%252Fip&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
> [junit4:junit4] 2> 110865 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 110867 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"6",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:37170__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:37170/_xpo/ip"}
> [junit4:junit4] 2> 110868 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 110869 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
> [junit4:junit4] 2> 110876 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110877 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110877 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110877 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110876 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110876 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 110876 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 111266 T325 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1
> [junit4:junit4] 2> 111267 T325 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 111268 T325 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 111269 T325 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 111271 T325 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1/'
> [junit4:junit4] 2> 111273 T325 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1/lib/README' to classloader
> [junit4:junit4] 2> 111273 T325 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 111317 T325 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 111352 T325 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 111354 T325 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 111361 T325 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 111705 T325 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 111712 T325 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 111714 T325 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 111736 T325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 111740 T325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 111743 T325 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 111744 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 111744 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 111745 T325 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 111746 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 111746 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 111746 T325 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 111746 T325 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371476841015/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/
> [junit4:junit4] 2> 111747 T325 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
> [junit4:junit4] 2> 111747 T325 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 111748 T325 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5
> [junit4:junit4] 2> 111748 T325 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index/
> [junit4:junit4] 2> 111748 T325 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 111748 T325 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index
> [junit4:junit4] 2> 111751 T325 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 111751 T325 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 111758 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 111759 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 111759 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 111760 T325 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 111760 T325 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 111761 T325 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 111761 T325 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 111761 T325 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 111762 T325 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 111764 T325 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 111767 T325 oass.SolrIndexSearcher.<init> Opening Searcher@2a9ecf43 main
> [junit4:junit4] 2> 111768 T325 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 111768 T325 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 111773 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2a9ecf43 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 111775 T325 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 111775 T325 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37170/_xpo/ip collection:collection1 shard:shard2
> [junit4:junit4] 2> 111778 T325 oasc.ZkController.register We are http://127.0.0.1:37170/_xpo/ip/collection1/ and leader is http://127.0.0.1:58042/_xpo/ip/collection1/
> [junit4:junit4] 2> 111778 T325 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37170/_xpo/ip
> [junit4:junit4] 2> 111778 T325 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 111779 T325 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C211 name=collection1 org.apache.solr.core.SolrCore@a0a3409 url=http://127.0.0.1:37170/_xpo/ip/collection1 node=127.0.0.1:37170__xpo%2Fip C211_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:37170__xpo%2Fip, base_url=http://127.0.0.1:37170/_xpo/ip}
> [junit4:junit4] 2> 111779 T327 C211 P37170 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 111780 T327 C211 P37170 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 111780 T325 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 111780 T327 C211 P37170 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 111780 T327 C211 P37170 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 111781 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 111781 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 111782 T327 C211 P37170 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 111782 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 111788 T268 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 111872 T217 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 111874 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49202
> [junit4:junit4] 2> 111875 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 111875 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 111876 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735
> [junit4:junit4] 2> 111876 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/solr.xml
> [junit4:junit4] 2> 111876 T217 oasc.CoreContainer.<init> New CoreContainer 1011980385
> [junit4:junit4] 2> 111877 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/'
> [junit4:junit4] 2> 111877 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/'
> [junit4:junit4] 2> 111924 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 111924 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 111924 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 111925 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 111925 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 111925 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 111926 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 111926 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 111926 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 111927 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 111930 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 111931 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
> [junit4:junit4] 2> 111931 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 111932 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 111934 T339 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@138c115f name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 111934 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 111936 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 111940 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 111941 T341 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7660cc00 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 111941 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 111945 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> ASYNC NEW_CORE C212 name=collection1 org.apache.solr.core.SolrCore@13931c0f url=http://127.0.0.1:59777/_xpo/ip/collection1 node=127.0.0.1:59777__xpo%2Fip C212_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:59777__xpo%2Fip, base_url=http://127.0.0.1:59777/_xpo/ip}
> [junit4:junit4] 2> 112315 T310 C212 P59777 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:55519/_xpo/ip/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4] 2> 112318 T310 C212 P59777 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
> [junit4:junit4] 2> 112323 T310 C212 P59777 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:59777/_xpo/ip START replicas=[http://127.0.0.1:55519/_xpo/ip/collection1/] nUpdates=100
> [junit4:junit4] 2> 112325 T310 C212 P59777 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4] 2> 112326 T310 C212 P59777 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4] 2> 112326 T310 C212 P59777 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4] 2> 112326 T310 C212 P59777 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4] 2> 112326 T310 C212 P59777 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4] 2> 112326 T310 C212 P59777 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:55519/_xpo/ip/collection1/. core=collection1
> [junit4:junit4] 2> 112326 T310 C212 P59777 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> ASYNC NEW_CORE C213 name=collection1 org.apache.solr.core.SolrCore@1e72bd20 url=http://127.0.0.1:55519/_xpo/ip/collection1 node=127.0.0.1:55519__xpo%2Fip C213_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:55519__xpo%2Fip, base_url=http://127.0.0.1:55519/_xpo/ip, leader=true}
> [junit4:junit4] 2> 112340 T253 C213 P55519 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
> [junit4:junit4] 2> 112340 T255 C213 P55519 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4] 2> 112343 T255 C213 P55519 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 112343 T255 C213 P55519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 112354 T255 C213 P55519 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty1/index,segFN=segments_2,generation=2}
> [junit4:junit4] 2> 112355 T255 C213 P55519 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
> [junit4:junit4] 2> 112356 T255 C213 P55519 oass.SolrIndexSearcher.<init> Opening Searcher@3c2686b8 realtime
> [junit4:junit4] 2> 112356 T255 C213 P55519 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4] 2> 112356 T255 C213 P55519 oasup.LogUpdateProcessor.finish [collection1] webapp=/_xpo/ip path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
> [junit4:junit4] 2> 112358 T310 C212 P59777 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 112358 T310 C212 P59777 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
> [junit4:junit4] 2> 112364 T310 C212 P59777 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4] 2> 112383 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 112385 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"6",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:37170__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:37170/_xpo/ip"}
> [junit4:junit4] 2> ASYNC NEW_CORE C214 name=collection1 org.apache.solr.core.SolrCore@1e72bd20 url=http://127.0.0.1:55519/_xpo/ip/collection1 node=127.0.0.1:55519__xpo%2Fip C214_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:55519__xpo%2Fip, base_url=http://127.0.0.1:55519/_xpo/ip, leader=true}
> [junit4:junit4] 2> 112385 T254 C214 P55519 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 112386 T254 C214 P55519 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=20
> [junit4:junit4] 2> ASYNC NEW_CORE C215 name=collection1 org.apache.solr.core.SolrCore@13931c0f url=http://127.0.0.1:59777/_xpo/ip/collection1 node=127.0.0.1:59777__xpo%2Fip C215_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:59777__xpo%2Fip, base_url=http://127.0.0.1:59777/_xpo/ip}
> [junit4:junit4] 2> 112387 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Master's generation: 2
> [junit4:junit4] 2> 112387 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
> [junit4:junit4] 2> 112388 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Starting replication process
> [junit4:junit4] 2> 112391 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112391 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112392 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112391 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112391 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112392 T253 C214 P55519 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=2
> [junit4:junit4] 2> 112392 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112392 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112394 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
> [junit4:junit4] 2> 112395 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112396 T310 C215 P59777 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344
> [junit4:junit4] 2> 112397 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f36923d fullCopy=false
> [junit4:junit4] 2> 112404 T255 C214 P55519 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=2
> [junit4:junit4] 2> 112409 T310 C215 P59777 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
> [junit4:junit4] 2> 112415 T310 C215 P59777 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
> [junit4:junit4] 2> 112416 T310 C215 P59777 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
> [junit4:junit4] 2> 112418 T310 C215 P59777 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index,segFN=segments_2,generation=2}
> [junit4:junit4] 2> 112419 T310 C215 P59777 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
> [junit4:junit4] 2> 112419 T310 C215 P59777 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
> [junit4:junit4] 2> 112420 T310 C215 P59777 oass.SolrIndexSearcher.<init> Opening Searcher@4c653706 main
> [junit4:junit4] 2> 112421 T309 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c653706 main{StandardDirectoryReader(segments_2:1:nrt)}
> [junit4:junit4] 2> 112421 T310 C215 P59777 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344;done=true>>]
> [junit4:junit4] 2> 112421 T310 C215 P59777 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344
> [junit4:junit4] 2> 112422 T310 C215 P59777 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty4/index.20130617204724344
> [junit4:junit4] 2> 112422 T310 C215 P59777 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4] 2> 112423 T310 C215 P59777 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4] 2> 112423 T310 C215 P59777 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 112423 T310 C215 P59777 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 112426 T310 C215 P59777 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4] 2> 112788 T268 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
> [junit4:junit4] 2> 112789 T268 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=6&state=recovering&nodeName=127.0.0.1:37170__xpo%252Fip&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
> [junit4:junit4] 2> 112949 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49202__xpo%2Fip
> [junit4:junit4] 2> 112950 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49202__xpo%2Fip
> [junit4:junit4] 2> 112952 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112952 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
> [junit4:junit4] 2> 112953 T275 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 112953 T341 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 112953 T307 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 112953 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 112954 T291 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 112954 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 112954 T259 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 112955 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 112953 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 112953 T246 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 112955 T324 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 112955 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 112955 T239 oascc.ZkStateReader$3.process Updating live nodes... (7)
> [junit4:junit4] 2> 112954 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 112962 T344 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 112962 T344 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 113901 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 113902 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"5",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard1",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:59777__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:59777/_xpo/ip"}
> [junit4:junit4] 2> 113905 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"7",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:49202__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:49202/_xpo/ip"}
> [junit4:junit4] 2> 113906 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 113906 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
> [junit4:junit4] 2> 113910 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 113910 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 113910 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 113910 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 113910 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 113910 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 113910 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 113910 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 113985 T344 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1
> [junit4:junit4] 2> 113985 T344 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 113986 T344 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 113986 T344 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 113988 T344 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1/'
> [junit4:junit4] 2> 113988 T344 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1/lib/README' to classloader
> [junit4:junit4] 2> 113989 T344 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 114019 T344 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 114049 T344 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 114051 T344 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 114057 T344 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 114363 T344 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 114369 T344 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 114371 T344 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 114391 T344 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 114395 T344 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 114398 T344 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 114399 T344 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 114400 T344 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 114400 T344 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 114401 T344 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 114401 T344 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 114402 T344 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 114402 T344 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371476843735/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/
> [junit4:junit4] 2> 114402 T344 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
> [junit4:junit4] 2> 114403 T344 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 114403 T344 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6
> [junit4:junit4] 2> 114403 T344 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/index/
> [junit4:junit4] 2> 114403 T344 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 114404 T344 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/index
> [junit4:junit4] 2> 114407 T344 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty6/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 114407 T344 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 114409 T344 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 114409 T344 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 114410 T344 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 114410 T344 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 114411 T344 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 114411 T344 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 114411 T344 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 114412 T344 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 114412 T344 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 114414 T344 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 114417 T344 oass.SolrIndexSearcher.<init> Opening Searcher@693b079f main
> [junit4:junit4] 2> 114418 T344 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 114418 T344 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 114422 T345 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@693b079f main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 114424 T344 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 114424 T344 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49202/_xpo/ip collection:collection1 shard:shard3
> [junit4:junit4] 2> 114428 T344 oasc.ZkController.register We are http://127.0.0.1:49202/_xpo/ip/collection1/ and leader is http://127.0.0.1:44496/_xpo/ip/collection1/
> [junit4:junit4] 2> 114428 T344 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49202/_xpo/ip
> [junit4:junit4] 2> 114428 T344 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 114429 T344 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C216 name=collection1 org.apache.solr.core.SolrCore@29709fba url=http://127.0.0.1:49202/_xpo/ip/collection1 node=127.0.0.1:49202__xpo%2Fip C216_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49202__xpo%2Fip, base_url=http://127.0.0.1:49202/_xpo/ip}
> [junit4:junit4] 2> 114429 T346 C216 P49202 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 114430 T346 C216 P49202 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 114430 T344 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 114430 T346 C216 P49202 oasc.ZkController.publish publishing core=collection1 state=recovering
> [junit4:junit4] 2> 114430 T346 C216 P49202 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 114431 T217 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
> [junit4:junit4] 2> 114432 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
> [junit4:junit4] 2> 114432 T346 C216 P49202 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 114432 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> 114440 T286 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true
> [junit4:junit4] 2> 114499 T217 oejs.Server.doStart jetty-8.1.10.v20130312
> [junit4:junit4] 2> 114501 T217 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47253
> [junit4:junit4] 2> 114502 T217 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
> [junit4:junit4] 2> 114502 T217 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
> [junit4:junit4] 2> 114502 T217 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386
> [junit4:junit4] 2> 114502 T217 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/solr.xml
> [junit4:junit4] 2> 114503 T217 oasc.CoreContainer.<init> New CoreContainer 428115399
> [junit4:junit4] 2> 114503 T217 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/'
> [junit4:junit4] 2> 114503 T217 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/'
> [junit4:junit4] 2> 114543 T217 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
> [junit4:junit4] 2> 114543 T217 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
> [junit4:junit4] 2> 114543 T217 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
> [junit4:junit4] 2> 114544 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
> [junit4:junit4] 2> 114544 T217 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
> [junit4:junit4] 2> 114544 T217 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
> [junit4:junit4] 2> 114545 T217 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
> [junit4:junit4] 2> 114545 T217 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
> [junit4:junit4] 2> 114545 T217 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
> [junit4:junit4] 2> 114546 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
> [junit4:junit4] 2> 114549 T217 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
> [junit4:junit4] 2> 114549 T217 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:35312/solr
> [junit4:junit4] 2> 114550 T217 oasc.ZkController.checkChrootPath zkHost includes chroot
> [junit4:junit4] 2> 114551 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 114552 T358 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@179fc2d2 name:ZooKeeperConnection Watcher:127.0.0.1:35312 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 114553 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 114556 T217 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
> [junit4:junit4] 2> 114559 T217 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
> [junit4:junit4] 2> 114561 T360 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dac7392 name:ZooKeeperConnection Watcher:127.0.0.1:35312/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
> [junit4:junit4] 2> 114561 T217 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
> [junit4:junit4] 2> 114568 T217 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
> [junit4:junit4] 2> ASYNC NEW_CORE C217 name=collection1 org.apache.solr.core.SolrCore@a0a3409 url=http://127.0.0.1:37170/_xpo/ip/collection1 node=127.0.0.1:37170__xpo%2Fip C217_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:37170__xpo%2Fip, base_url=http://127.0.0.1:37170/_xpo/ip}
> [junit4:junit4] 2> 114790 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:58042/_xpo/ip/collection1/ core=collection1 - recoveringAfterStartup=true
> [junit4:junit4] 2> 114790 T327 C217 P37170 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:37170/_xpo/ip START replicas=[http://127.0.0.1:58042/_xpo/ip/collection1/] nUpdates=100
> [junit4:junit4] 2> 114791 T327 C217 P37170 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
> [junit4:junit4] 2> 114791 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
> [junit4:junit4] 2> 114791 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
> [junit4:junit4] 2> 114791 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
> [junit4:junit4] 2> 114791 T327 C217 P37170 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
> [junit4:junit4] 2> 114792 T327 C217 P37170 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:58042/_xpo/ip/collection1/. core=collection1
> [junit4:junit4] 2> 114792 T327 C217 P37170 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> [junit4:junit4] 2> ASYNC NEW_CORE C218 name=collection1 org.apache.solr.core.SolrCore@7633a644 url=http://127.0.0.1:58042/_xpo/ip/collection1 node=127.0.0.1:58042__xpo%2Fip C218_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:58042__xpo%2Fip, base_url=http://127.0.0.1:58042/_xpo/ip, leader=true}
> [junit4:junit4] 2> 114800 T268 C218 P58042 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
> [junit4:junit4] 2> 114800 T271 C218 P58042 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4:junit4] 2> 114805 T271 C218 P58042 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 114806 T271 C218 P58042 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 114815 T271 C218 P58042 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty2/index,segFN=segments_2,generation=2}
> [junit4:junit4] 2> 114815 T271 C218 P58042 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
> [junit4:junit4] 2> 114815 T271 C218 P58042 oass.SolrIndexSearcher.<init> Opening Searcher@593c82ea realtime
> [junit4:junit4] 2> 114816 T271 C218 P58042 oasu.DirectUpdateHandler2.commit end_commit_flush
> [junit4:junit4] 2> 114816 T271 C218 P58042 oasup.LogUpdateProcessor.finish [collection1] webapp=/_xpo/ip path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
> [junit4:junit4] 2> 114817 T327 C217 P37170 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 114817 T327 C217 P37170 oash.SnapPuller.<init> No value set for 'pollInterval'. Timer Task not started.
> [junit4:junit4] 2> 114820 T268 C218 P58042 oash.ReplicationHandler.inform Commits will be reserved for 10000
> [junit4:junit4] 2> 114820 T268 C218 P58042 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
> [junit4:junit4] 2> 114821 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Master's generation: 2
> [junit4:junit4] 2> 114822 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
> [junit4:junit4] 2> 114822 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Starting replication process
> [junit4:junit4] 2> 114824 T271 C218 P58042 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
> [junit4:junit4] 2> 114825 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
> [junit4:junit4] 2> 114826 T327 C217 P37170 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774
> [junit4:junit4] 2> 114826 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7cf2a936 fullCopy=false
> [junit4:junit4] 2> 114829 T270 C218 P58042 oasc.SolrCore.execute [collection1] webapp=/_xpo/ip path=/replication params={file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
> [junit4:junit4] 2> 114830 T327 C217 P37170 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
> [junit4:junit4] 2> 114835 T327 C217 P37170 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
> [junit4:junit4] 2> 114835 T327 C217 P37170 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
> [junit4:junit4] 2> 114837 T327 C217 P37170 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index,segFN=segments_2,generation=2}
> [junit4:junit4] 2> 114837 T327 C217 P37170 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
> [junit4:junit4] 2> 114838 T327 C217 P37170 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
> [junit4:junit4] 2> 114838 T327 C217 P37170 oass.SolrIndexSearcher.<init> Opening Searcher@550371aa main
> [junit4:junit4] 2> 114839 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@550371aa main{StandardDirectoryReader(segments_2:1:nrt)}
> [junit4:junit4] 2> 114839 T327 C217 P37170 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774;done=true>>]
> [junit4:junit4] 2> 114839 T327 C217 P37170 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774
> [junit4:junit4] 2> 114840 T327 C217 P37170 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty5/index.20130617204726774
> [junit4:junit4] 2> 114840 T327 C217 P37170 oasc.RecoveryStrategy.replay No replay needed. core=collection1
> [junit4:junit4] 2> 114840 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
> [junit4:junit4] 2> 114840 T327 C217 P37170 oasc.ZkController.publish publishing core=collection1 state=active
> [junit4:junit4] 2> 114840 T327 C217 P37170 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 114842 T327 C217 P37170 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
> [junit4:junit4] 2> 115419 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 115420 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"7",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard3",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"recovering",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:49202__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:49202/_xpo/ip"}
> [junit4:junit4] 2> 115423 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"6",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"active",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:37170__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:37170/_xpo/ip"}
> [junit4:junit4] 2> 115445 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115445 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115445 T360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115445 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115445 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115445 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115445 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115445 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115445 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115573 T217 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47253__xpo%2Fip
> [junit4:junit4] 2> 115574 T217 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47253__xpo%2Fip
> [junit4:junit4] 2> 115576 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115576 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
> [junit4:junit4] 2> 115577 T341 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115577 T275 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115577 T259 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115578 T307 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115578 T246 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115577 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 115578 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 115578 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 115578 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 115578 T291 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115578 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 115579 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 115581 T360 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115581 T360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 115581 T239 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115582 T324 oascc.ZkStateReader$3.process Updating live nodes... (8)
> [junit4:junit4] 2> 115589 T363 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 115589 T363 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 116441 T286 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 7, state: recovering, checkLive: true, onlyIfLeader: true for: 2 seconds.
> [junit4:junit4] 2> 116441 T286 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=7&state=recovering&nodeName=127.0.0.1:49202__xpo%252Fip&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2001
> [junit4:junit4] 2> 116955 T240 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 116956 T240 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"8",
> [junit4:junit4] 2> "numShards":"3",
> [junit4:junit4] 2> "shard":null,
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:47253__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:47253/_xpo/ip"}
> [junit4:junit4] 2> 116956 T240 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
> [junit4:junit4] 2> 116957 T240 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
> [junit4:junit4] 2> 116963 T239 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 116963 T341 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 116963 T259 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 116963 T291 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 116963 T360 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 116963 T246 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 116963 T307 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 116963 T275 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 116963 T324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
> [junit4:junit4] 2> 117590 T363 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1
> [junit4:junit4] 2> 117591 T363 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
> [junit4:junit4] 2> 117592 T363 oasc.ZkController.createCollectionZkNode Collection zkNode exists
> [junit4:junit4] 2> 117592 T363 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
> [junit4:junit4] 2> 117593 T363 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1/'
> [junit4:junit4] 2> 117594 T363 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1/lib/README' to classloader
> [junit4:junit4] 2> 117594 T363 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1/lib/classes/' to classloader
> [junit4:junit4] 2> 117624 T363 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_44
> [junit4:junit4] 2> 117658 T363 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
> [junit4:junit4] 2> 117660 T363 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
> [junit4:junit4] 2> 117668 T363 oass.IndexSchema.readSchema [collection1] Schema name=test
> [junit4:junit4] 2> 118003 T363 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
> [junit4:junit4] 2> 118009 T363 oass.IndexSchema.readSchema default search field in schema is text
> [junit4:junit4] 2> 118011 T363 oass.IndexSchema.readSchema unique key field: id
> [junit4:junit4] 2> 118036 T363 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 118041 T363 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
> [junit4:junit4] 2> 118046 T363 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 118048 T363 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 118048 T363 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 118049 T363 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
> [junit4:junit4] 2> 118050 T363 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
> [junit4:junit4] 2> 118050 T363 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
> [junit4:junit4] 2> 118051 T363 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 118051 T363 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371476846386/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/
> [junit4:junit4] 2> 118051 T363 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@56d54467
> [junit4:junit4] 2> 118052 T363 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
> [junit4:junit4] 2> 118053 T363 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7
> [junit4:junit4] 2> 118053 T363 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/index/
> [junit4:junit4] 2> 118054 T363 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/index' doesn't exist. Creating new index...
> [junit4:junit4] 2> 118054 T363 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/index
> [junit4:junit4] 2> 118058 T363 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
> [junit4:junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty7/index,segFN=segments_1,generation=1}
> [junit4:junit4] 2> 118058 T363 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
> [junit4:junit4] 2> 118061 T363 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
> [junit4:junit4] 2> 118061 T363 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
> [junit4:junit4] 2> 118062 T363 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
> [junit4:junit4] 2> 118063 T363 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
> [junit4:junit4] 2> 118063 T363 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
> [junit4:junit4] 2> 118063 T363 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
> [junit4:junit4] 2> 118064 T363 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
> [junit4:junit4] 2> 118064 T363 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
> [junit4:junit4] 2> 118065 T363 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
> [junit4:junit4] 2> 118069 T363 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
> [junit4:junit4] 2> 118073 T363 oass.SolrIndexSearcher.<init> Opening Searcher@57fef174 main
> [junit4:junit4] 2> 118074 T363 oasu.CommitTracker.<init> Hard AutoCommit: disabled
> [junit4:junit4] 2> 118074 T363 oasu.CommitTracker.<init> Soft AutoCommit: disabled
> [junit4:junit4] 2> 118079 T364 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@57fef174 main{StandardDirectoryReader(segments_1:1)}
> [junit4:junit4] 2> 118080 T363 oasc.CoreContainer.registerCore registering core: collection1
> [junit4:junit4] 2> 118081 T363 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47253/_xpo/ip collection:collection1 shard:shard1
> [junit4:junit4] 2> 118085 T363 oasc.ZkController.register We are http://127.0.0.1:47253/_xpo/ip/collection1/ and leader is http://127.0.0.1:55519/_xpo/ip/collection1/
> [junit4:junit4] 2> 118085 T363 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47253/_xpo/ip
> [junit4:junit4] 2> 118085 T363 oasc.ZkController.checkRecovery Core needs to recover:collection1
> [junit4:junit4] 2> 118085 T363 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
> [junit4:junit4] 2> ASYNC NEW_CORE C219 name=collection1 org.apache.solr.core.SolrCore@2cfa3e0e url=http://127.0.0.1:47253/_xpo/ip/collection1 node=127.0.0.1:47253__xpo%2Fip C219_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:47253__xpo%2Fip, base_url=http://127.0.0.1:47253/_xpo/ip}
> [junit4:junit4] 2> 118086 T365 C219 P47253 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
> [junit4:junit4] 2> 118086 T365 C219 P47253 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
> [junit4:junit4] 2> 118086 T363
>
> [...truncated too long message...]
>
> the main queue
> [junit4:junit4] 2> 346295 T715 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
> [junit4:junit4] 2> 346296 T715 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"12",
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "node_name":"127.0.0.1:40497__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:40497/_xpo/ip"}
> [junit4:junit4] 2> 346299 T715 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
> [junit4:junit4] 2> "operation":"state",
> [junit4:junit4] 2> "core_node_name":"12",
> [junit4:junit4] 2> "numShards":null,
> [junit4:junit4] 2> "shard":"shard2",
> [junit4:junit4] 2> "roles":null,
> [junit4:junit4] 2> "state":"down",
> [junit4:junit4] 2> "shard_state":"active",
> [junit4:junit4] 2> "core":"collection1",
> [junit4:junit4] 2> "collection":"collection1",
> [junit4:junit4] 2> "shard_range":null,
> [junit4:junit4] 2> "node_name":"127.0.0.1:40497__xpo%2Fip",
> [junit4:junit4] 2> "base_url":"http://127.0.0.1:40497/_xpo/ip"}
> [junit4:junit4] 2> 346305 T457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 346307 T217 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_xpo/ip,null}
> [junit4:junit4] 2> 346359 T217 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 37443
> [junit4:junit4] 2> 346360 T217 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=695679923
> [junit4:junit4] 2> 346498 T419 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4:junit4] 2> 346498 T419 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
> [junit4:junit4] 2> 346498 T419 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4:junit4] 2> 347361 T217 oasc.ZkController.publish publishing core=collection1 state=down
> [junit4:junit4] 2> 347362 T217 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
> [junit4:junit4] 2> 347364 T217 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@287ab2c
> [junit4:junit4] 2> 347376 T217 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=22,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=1242,cumulative_deletesById=596,cumulative_deletesByQuery=1,cumulative_errors=0}
> [junit4:junit4] 2> 347376 T217 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
> [junit4:junit4] 2> 347377 T217 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
> [junit4:junit4] 2> 347377 T217 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
> [junit4:junit4] 2> 347378 T217 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
> [junit4:junit4] 2> 347379 T217 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
> [junit4:junit4] 2> 347379 T217 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12;done=false>>]
> [junit4:junit4] 2> 347379 T217 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12
> [junit4:junit4] 2> 347379 T217 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12/index;done=false>>]
> [junit4:junit4] 2> 347380 T217 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371476825183/jetty12/index
> [junit4:junit4] 2> 347381 T715 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89881105220763676-127.0.0.1:37443__xpo%2Fip-n_0000000012) am no longer a leader.
> [junit4:junit4] 2> 347404 T217 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_xpo/ip,null}
> [junit4:junit4] 2> 347473 T217 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
> [junit4:junit4] 2> 347475 T217 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:35312 35312
> [junit4:junit4] 1> client port:0.0.0.0/0.0.0.0:0
> [junit4:junit4] 1> safe stop:
> [junit4:junit4] 1> safe stop:
> [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=62718A172099C606 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_CS -Dtests.timezone=Antarctica/Davis -Dtests.file.encoding=ISO-8859-1
> [junit4:junit4] FAILURE 254s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
> [junit4:junit4] > Throwable #1: java.lang.AssertionError: shard3 is not consistent. Got 648 from http://127.0.0.1:49202/_xpo/ip/collection1lastClient and got 646 from http://127.0.0.1:37443/_xpo/ip/collection1
> [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([62718A172099C606:E397040F57C6A63A]:0)
> [junit4:junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
> [junit4:junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
> [junit4:junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
> [junit4:junit4] > at java.lang.Thread.run(Thread.java:722)
> [junit4:junit4] 2> 347520 T217 oas.SolrTestCaseJ4.deleteCore ###deleteCore
> [junit4:junit4] 2> 254309 T216 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
> [junit4:junit4] 2> 347585 T436 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4:junit4] 2> 347586 T436 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
> [junit4:junit4] 2> 347586 T436 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4:junit4] 2> 348682 T457 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4:junit4] 2> 348682 T457 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
> [junit4:junit4] 2> 348682 T457 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
> [junit4:junit4] 2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=324), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=324)), sim=DefaultSimilarity, locale=sr_CS, timezone=Antarctica/Davis
> [junit4:junit4] 2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_21 (64-bit)/cpus=8,threads=4,free=89012592,total=317915136
> [junit4:junit4] 2> NOTE: All tests run in this JVM: [TestUpdate, BadIndexSchemaTest, IndexBasedSpellCheckerTest, TestQuerySenderNoQuery, NumericFieldsTest, TestRandomFaceting, TestFieldTypeResource, HighlighterConfigTest, TestAddFieldRealTimeGet, TestSolrQueryParserDefaultOperatorResource, DirectSolrSpellCheckerTest, ChaosMonkeySafeLeaderTest]
> [junit4:junit4] Completed on J1 in 255.60s, 1 test, 1 failure <<< FAILURES!
>
> [...truncated 816 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:392: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:372: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1246: The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:890: There were test failures: 298 suites, 1252 tests, 1 failure, 14 ignored (8 assumptions)
>
> Total time: 39 minutes 37 seconds
> Build step 'Invoke Ant' marked build as failure
> Description set: Java: 64bit/jdk1.7.0_21 -XX:-UseCompressedOops -XX:+UseParallelGC
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org