You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/10/21 02:45:27 UTC
[JENKINS] Lucene-Solr-4.x-MacOSX (64bit/jdk1.7.0) - Build # 907 -
Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-MacOSX/907/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:53365/ui_s/collection1, http://127.0.0.1:53324/ui_s/collection1, http://127.0.0.1:53331/ui_s/collection1, http://127.0.0.1:53317/ui_s/collection1, http://127.0.0.1:53314/ui_s/collection1, http://127.0.0.1:53311/ui_s/collection1, http://127.0.0.1:53350/ui_s/collection1, http://127.0.0.1:53355/ui_s/collection1, http://127.0.0.1:53338/ui_s/collection1, http://127.0.0.1:53320/ui_s/collection1, http://127.0.0.1:53360/ui_s/collection1, http://127.0.0.1:53345/ui_s/collection1]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:53365/ui_s/collection1, http://127.0.0.1:53324/ui_s/collection1, http://127.0.0.1:53331/ui_s/collection1, http://127.0.0.1:53317/ui_s/collection1, http://127.0.0.1:53314/ui_s/collection1, http://127.0.0.1:53311/ui_s/collection1, http://127.0.0.1:53350/ui_s/collection1, http://127.0.0.1:53355/ui_s/collection1, http://127.0.0.1:53338/ui_s/collection1, http://127.0.0.1:53320/ui_s/collection1, http://127.0.0.1:53360/ui_s/collection1, http://127.0.0.1:53345/ui_s/collection1]
at __randomizedtesting.SeedInfo.seed([79D554505CB7677B:F833DA482BE80747]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:337)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:636)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1140)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:138)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:744)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:53338/ui_s/collection1
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:476)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
... 46 more
Caused by: java.net.SocketException: Invalid argument
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:366)
... 48 more
Build Log:
[...truncated 10246 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> 1803402 T4362 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /ui_s/
[junit4] 2> 1803413 T4362 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./solrtest-ChaosMonkeySafeLeaderTest-1382314324510
[junit4] 2> 1803415 T4362 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1803416 T4363 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1803523 T4362 oasc.ZkTestServer.run start zk server on port:53304
[junit4] 2> 1803526 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1803541 T4369 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5e86a9dd name:ZooKeeperConnection Watcher:127.0.0.1:53304 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1803542 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1803542 T4362 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1803556 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1803565 T4371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2f6baf11 name:ZooKeeperConnection Watcher:127.0.0.1:53304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1803566 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1803568 T4362 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1803575 T4362 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1803583 T4362 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1803589 T4362 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1803596 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1803598 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1803611 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1803612 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1803621 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1803622 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1803630 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1803631 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1803640 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1803641 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1803650 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1803651 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1803658 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1803659 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1803667 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1803668 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1803676 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1803677 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1803689 T4362 oasc.AbstractZkTestCase.putConfig put /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1803690 T4362 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1804129 T4362 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1804136 T4362 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53307
[junit4] 2> 1804139 T4362 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1804139 T4362 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1804140 T4362 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793
[junit4] 2> 1804142 T4362 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793/'
[junit4] 2> 1804184 T4362 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793/solr.xml
[junit4] 2> 1804297 T4362 oasc.CoreContainer.<init> New CoreContainer 1257947028
[junit4] 2> 1804298 T4362 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793/]
[junit4] 2> 1804300 T4362 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1804300 T4362 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1804301 T4362 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1804301 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1804302 T4362 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1804302 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1804302 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1804303 T4362 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1804303 T4362 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1804309 T4362 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1804310 T4362 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1804311 T4362 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53304/solr
[junit4] 2> 1804311 T4362 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1804313 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1804317 T4382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32b77be1 name:ZooKeeperConnection Watcher:127.0.0.1:53304 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1804318 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1804324 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1804328 T4384 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58485703 name:ZooKeeperConnection Watcher:127.0.0.1:53304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1804329 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1804333 T4362 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1804341 T4362 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1804349 T4362 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1804354 T4362 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53307_ui_s
[junit4] 2> 1804356 T4362 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53307_ui_s
[junit4] 2> 1804364 T4362 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1804377 T4362 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1804383 T4362 oasc.Overseer.start Overseer (id=90591351571677187-127.0.0.1:53307_ui_s-n_0000000000) starting
[junit4] 2> 1804391 T4362 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1804401 T4386 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1804403 T4362 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1804408 T4362 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1804413 T4362 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1804420 T4385 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1804432 T4387 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1804432 T4387 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1804435 T4387 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1805938 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1805942 T4385 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:53307/ui_s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53307_ui_s",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1805943 T4385 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1805945 T4385 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1805971 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1806437 T4387 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1806439 T4387 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793/collection1
[junit4] 2> 1806440 T4387 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1806443 T4387 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1806444 T4387 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1806450 T4387 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793/collection1/'
[junit4] 2> 1806455 T4387 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793/collection1/lib/classes/' to classloader
[junit4] 2> 1806457 T4387 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793/collection1/lib/README' to classloader
[junit4] 2> 1806582 T4387 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
[junit4] 2> 1806671 T4387 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1806675 T4387 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1806689 T4387 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1807612 T4387 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 1807613 T4387 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 1807614 T4387 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1807628 T4387 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1807635 T4387 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1807665 T4387 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1807670 T4387 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1807676 T4387 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1807678 T4387 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1807678 T4387 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1807679 T4387 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1807681 T4387 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1807681 T4387 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1807682 T4387 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1807682 T4387 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1382314324793/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/control/data/
[junit4] 2> 1807682 T4387 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@151418b5
[junit4] 2> 1807684 T4387 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/control/data
[junit4] 2> 1807685 T4387 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/control/data/index/
[junit4] 2> 1807685 T4387 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1807686 T4387 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/control/data/index
[junit4] 2> 1807687 T4387 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=40.333984375, floorSegmentMB=1.73828125, forceMergeDeletesPctAllowed=3.0920996602059603, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1807692 T4387 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1807692 T4387 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1807697 T4387 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1807698 T4387 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1807699 T4387 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1807699 T4387 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1807699 T4387 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1807700 T4387 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1807701 T4387 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1807701 T4387 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1807702 T4387 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1807702 T4387 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1807703 T4387 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1807703 T4387 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1807704 T4387 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1807705 T4387 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1807705 T4387 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1807720 T4387 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1807725 T4387 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1807725 T4387 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1807726 T4387 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=79.3193359375, floorSegmentMB=2.1640625, forceMergeDeletesPctAllowed=14.853468200392463, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4683090916439746
[junit4] 2> 1807728 T4387 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1807729 T4387 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1807729 T4387 oass.SolrIndexSearcher.<init> Opening Searcher@3b22ef96 main
[junit4] 2> 1807733 T4388 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b22ef96 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1807736 T4387 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1807736 T4387 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53307/ui_s collection:control_collection shard:shard1
[junit4] 2> 1807739 T4387 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1807755 T4387 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1807760 T4387 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1807761 T4387 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1807761 T4387 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53307/ui_s/collection1/
[junit4] 2> 1807761 T4387 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1807762 T4387 oasc.SyncStrategy.syncToMe http://127.0.0.1:53307/ui_s/collection1/ has no replicas
[junit4] 2> 1807762 T4387 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53307/ui_s/collection1/ shard1
[junit4] 2> 1807762 T4387 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1808997 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1809037 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1809077 T4387 oasc.ZkController.register We are http://127.0.0.1:53307/ui_s/collection1/ and leader is http://127.0.0.1:53307/ui_s/collection1/
[junit4] 2> 1809078 T4387 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53307/ui_s
[junit4] 2> 1809079 T4387 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1809079 T4387 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1809080 T4387 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1809083 T4387 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1809086 T4362 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1809086 T4362 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1809090 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1809095 T4391 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f91810 name:ZooKeeperConnection Watcher:127.0.0.1:53304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1809095 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1809098 T4362 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1809103 T4362 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4] 2> 1809541 T4362 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1809548 T4362 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53311
[junit4] 2> 1809549 T4362 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1809550 T4362 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1809551 T4362 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198
[junit4] 2> 1809551 T4362 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198/'
[junit4] 2> 1809594 T4362 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198/solr.xml
[junit4] 2> 1809716 T4362 oasc.CoreContainer.<init> New CoreContainer 1657369839
[junit4] 2> 1809717 T4362 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198/]
[junit4] 2> 1809724 T4362 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1809724 T4362 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1809725 T4362 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1809726 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1809726 T4362 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1809727 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1809728 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1809728 T4362 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1809729 T4362 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1809737 T4362 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1809737 T4362 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1809738 T4362 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53304/solr
[junit4] 2> 1809739 T4362 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1809740 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1809746 T4402 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18d28bb6 name:ZooKeeperConnection Watcher:127.0.0.1:53304 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1809747 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1809754 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1809762 T4404 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@211fff53 name:ZooKeeperConnection Watcher:127.0.0.1:53304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1809763 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1809772 T4362 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1810574 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1810578 T4385 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:53307/ui_s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53307_ui_s",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1810596 T4391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1810597 T4404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1810597 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1810783 T4362 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53311_ui_s
[junit4] 2> 1810788 T4362 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53311_ui_s
[junit4] 2> 1810800 T4384 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1810801 T4404 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1810801 T4391 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1810818 T4405 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1810818 T4405 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1810821 T4405 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1812119 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1812123 T4385 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:53311/ui_s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53311_ui_s",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1812124 T4385 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 1812125 T4385 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1812146 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1812146 T4391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1812146 T4404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1812825 T4405 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1812825 T4405 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198/collection1
[junit4] 2> 1812826 T4405 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1812828 T4405 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1812829 T4405 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1812834 T4405 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198/collection1/'
[junit4] 2> 1812838 T4405 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198/collection1/lib/classes/' to classloader
[junit4] 2> 1812840 T4405 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198/collection1/lib/README' to classloader
[junit4] 2> 1812943 T4405 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
[junit4] 2> 1813042 T4405 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1813046 T4405 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1813058 T4405 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1813930 T4405 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 1813932 T4405 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 1813933 T4405 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1813948 T4405 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1813953 T4405 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1813985 T4405 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1813992 T4405 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1813999 T4405 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1814001 T4405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1814002 T4405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1814002 T4405 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1814005 T4405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1814005 T4405 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1814006 T4405 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1814006 T4405 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1382314330198/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty1/
[junit4] 2> 1814007 T4405 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@151418b5
[junit4] 2> 1814008 T4405 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty1
[junit4] 2> 1814009 T4405 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty1/index/
[junit4] 2> 1814010 T4405 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1814010 T4405 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty1/index
[junit4] 2> 1814011 T4405 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=40.333984375, floorSegmentMB=1.73828125, forceMergeDeletesPctAllowed=3.0920996602059603, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1814016 T4405 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1814016 T4405 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1814023 T4405 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1814024 T4405 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1814024 T4405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1814025 T4405 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1814025 T4405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1814025 T4405 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1814026 T4405 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1814027 T4405 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1814027 T4405 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1814028 T4405 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1814029 T4405 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1814029 T4405 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1814030 T4405 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1814030 T4405 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1814031 T4405 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1814045 T4405 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1814051 T4405 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1814051 T4405 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1814053 T4405 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=79.3193359375, floorSegmentMB=2.1640625, forceMergeDeletesPctAllowed=14.853468200392463, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4683090916439746
[junit4] 2> 1814056 T4405 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1814056 T4405 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1814057 T4405 oass.SolrIndexSearcher.<init> Opening Searcher@4c941c4f main
[junit4] 2> 1814077 T4406 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4c941c4f main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1814079 T4405 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1814079 T4405 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53311/ui_s collection:collection1 shard:shard1
[junit4] 2> 1814082 T4405 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1814104 T4405 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1814111 T4405 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1814111 T4405 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1814112 T4405 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53311/ui_s/collection1/
[junit4] 2> 1814112 T4405 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1814113 T4405 oasc.SyncStrategy.syncToMe http://127.0.0.1:53311/ui_s/collection1/ has no replicas
[junit4] 2> 1814113 T4405 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53311/ui_s/collection1/ shard1
[junit4] 2> 1814113 T4405 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1815177 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1815214 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1815214 T4391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1815215 T4404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1815248 T4405 oasc.ZkController.register We are http://127.0.0.1:53311/ui_s/collection1/ and leader is http://127.0.0.1:53311/ui_s/collection1/
[junit4] 2> 1815249 T4405 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53311/ui_s
[junit4] 2> 1815250 T4405 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1815251 T4405 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1815252 T4405 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1815261 T4405 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1815268 T4362 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1815269 T4362 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1815730 T4362 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1815734 T4362 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53314
[junit4] 2> 1815736 T4362 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1815736 T4362 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1815737 T4362 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369
[junit4] 2> 1815737 T4362 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369/'
[junit4] 2> 1815775 T4362 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369/solr.xml
[junit4] 2> 1815883 T4362 oasc.CoreContainer.<init> New CoreContainer 204151839
[junit4] 2> 1815883 T4362 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369/]
[junit4] 2> 1815886 T4362 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1815886 T4362 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1815888 T4362 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1815888 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1815889 T4362 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1815889 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1815890 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1815890 T4362 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1815890 T4362 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1815896 T4362 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1815896 T4362 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1815897 T4362 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53304/solr
[junit4] 2> 1815897 T4362 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1815899 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1815903 T4418 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@aca756f name:ZooKeeperConnection Watcher:127.0.0.1:53304 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1815905 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1815910 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1815914 T4420 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e1ddfa0 name:ZooKeeperConnection Watcher:127.0.0.1:53304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1815915 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1815923 T4362 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1816754 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1816757 T4385 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:53311/ui_s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53311_ui_s",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1816778 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1816780 T4420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1816779 T4404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1816779 T4391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1816934 T4362 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53314_ui_s
[junit4] 2> 1816939 T4362 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53314_ui_s
[junit4] 2> 1816955 T4420 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1816957 T4384 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1816957 T4391 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1816958 T4404 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1816989 T4421 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1816989 T4421 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1816992 T4421 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1818308 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1818312 T4385 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:53314/ui_s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53314_ui_s",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1818313 T4385 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1818314 T4385 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1818337 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1818338 T4391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1818340 T4404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1818338 T4420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1818997 T4421 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1818998 T4421 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369/collection1
[junit4] 2> 1818999 T4421 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1819003 T4421 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1819003 T4421 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1819009 T4421 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369/collection1/'
[junit4] 2> 1819013 T4421 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369/collection1/lib/classes/' to classloader
[junit4] 2> 1819015 T4421 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369/collection1/lib/README' to classloader
[junit4] 2> 1819165 T4421 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
[junit4] 2> 1819254 T4421 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1819258 T4421 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1819271 T4421 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1820150 T4421 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 1820151 T4421 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 1820152 T4421 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1820164 T4421 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1820169 T4421 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1820204 T4421 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1820212 T4421 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1820220 T4421 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1820222 T4421 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1820223 T4421 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1820223 T4421 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1820226 T4421 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1820226 T4421 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1820227 T4421 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1820227 T4421 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1382314336369/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty2/
[junit4] 2> 1820227 T4421 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@151418b5
[junit4] 2> 1820229 T4421 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty2
[junit4] 2> 1820230 T4421 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty2/index/
[junit4] 2> 1820231 T4421 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1820232 T4421 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty2/index
[junit4] 2> 1820233 T4421 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=40.333984375, floorSegmentMB=1.73828125, forceMergeDeletesPctAllowed=3.0920996602059603, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1820237 T4421 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1820237 T4421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1820244 T4421 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1820245 T4421 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1820246 T4421 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1820246 T4421 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1820247 T4421 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1820247 T4421 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1820248 T4421 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1820249 T4421 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1820249 T4421 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1820251 T4421 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1820252 T4421 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1820252 T4421 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1820253 T4421 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1820254 T4421 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1820255 T4421 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1820269 T4421 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1820274 T4421 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1820274 T4421 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1820276 T4421 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=45, maxMergedSegmentMB=79.3193359375, floorSegmentMB=2.1640625, forceMergeDeletesPctAllowed=14.853468200392463, segmentsPerTier=41.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4683090916439746
[junit4] 2> 1820279 T4421 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1820279 T4421 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1820279 T4421 oass.SolrIndexSearcher.<init> Opening Searcher@49e3c6d main
[junit4] 2> 1820286 T4422 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@49e3c6d main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1820289 T4421 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1820289 T4421 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53314/ui_s collection:collection1 shard:shard2
[junit4] 2> 1820291 T4421 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1820307 T4421 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1820314 T4421 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1820314 T4421 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1820314 T4421 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53314/ui_s/collection1/
[junit4] 2> 1820315 T4421 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1820315 T4421 oasc.SyncStrategy.syncToMe http://127.0.0.1:53314/ui_s/collection1/ has no replicas
[junit4] 2> 1820315 T4421 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53314/ui_s/collection1/ shard2
[junit4] 2> 1820315 T4421 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1821373 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1821416 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1821417 T4404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1821416 T4391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1821418 T4420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1821466 T4421 oasc.ZkController.register We are http://127.0.0.1:53314/ui_s/collection1/ and leader is http://127.0.0.1:53314/ui_s/collection1/
[junit4] 2> 1821466 T4421 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53314/ui_s
[junit4] 2> 1821467 T4421 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1821467 T4421 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1821468 T4421 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1821473 T4421 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1821481 T4362 oass.SolrDispatchFilter.init user.dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0
[junit4] 2> 1821482 T4362 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1821926 T4362 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1821931 T4362 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53317
[junit4] 2> 1821932 T4362 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1821933 T4362 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1821933 T4362 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580
[junit4] 2> 1821933 T4362 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580/'
[junit4] 2> 1821969 T4362 oasc.ConfigSolr.fromFile Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580/solr.xml
[junit4] 2> 1822092 T4362 oasc.CoreContainer.<init> New CoreContainer 519329327
[junit4] 2> 1822093 T4362 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580/]
[junit4] 2> 1822095 T4362 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4] 2> 1822095 T4362 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1822096 T4362 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1822096 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1822096 T4362 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1822097 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1822097 T4362 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1822098 T4362 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1822098 T4362 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1822104 T4362 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1822105 T4362 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1822105 T4362 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53304/solr
[junit4] 2> 1822106 T4362 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1822107 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1822113 T4434 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a014d9a name:ZooKeeperConnection Watcher:127.0.0.1:53304 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1822114 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1822120 T4362 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1822126 T4436 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@72544084 name:ZooKeeperConnection Watcher:127.0.0.1:53304/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1822126 T4362 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1822136 T4362 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1822948 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1822952 T4385 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:53314/ui_s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53314_ui_s",
[junit4] 2> "shard":"shard2",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1822967 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1822967 T4436 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1822968 T4404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1822969 T4420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1822968 T4391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1823146 T4362 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53317_ui_s
[junit4] 2> 1823149 T4362 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53317_ui_s
[junit4] 2> 1823157 T4436 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1823158 T4384 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1823158 T4391 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1823158 T4420 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1823158 T4404 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1823178 T4437 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1823179 T4437 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1823183 T4437 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1824486 T4385 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1824489 T4385 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:53317/ui_s",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:53317_ui_s",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node3"}
[junit4] 2> 1824490 T4385 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 1824491 T4385 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 1824503 T4384 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1824503 T4391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1824504 T4436 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1824505 T4404 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1824504 T4420 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1825187 T4437 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1825187 T4437 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580/collection1
[junit4] 2> 1825188 T4437 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1825190 T4437 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1825190 T4437 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1825193 T4437 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580/collection1/'
[junit4] 2> 1825196 T4437 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580/collection1/lib/classes/' to classloader
[junit4] 2> 1825197 T4437 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580/collection1/lib/README' to classloader
[junit4] 2> 1825274 T4437 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_46
[junit4] 2> 1825375 T4437 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1825378 T4437 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1825393 T4437 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1826271 T4437 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 1826273 T4437 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 1826274 T4437 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1826286 T4437 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1826290 T4437 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1826320 T4437 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1826328 T4437 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1826336 T4437 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1826338 T4437 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1826339 T4437 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1826339 T4437 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1826342 T4437 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1826342 T4437 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1826343 T4437 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1826344 T4437 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1382314342580/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty3/
[junit4] 2> 1826344 T4437 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@151418b5
[junit4] 2> 1826345 T4437 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty3
[junit4] 2> 1826346 T4437 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty3/index/
[junit4] 2> 1826347 T4437 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 1826348 T4437 oasc.CachingDirectoryFactory.get return new directory for /Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty3/index
[junit4] 2> 1826348 T4437 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=14, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=40.333984375, floorSegmentMB=1.73828125, forceMergeDeletesPctAllowed=3.0920996602059603, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1826353 T4437 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build/solr-core/test/J0/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1382314324509/jetty3/index,segFN=segments_1,generation=1}
[junit4] 2> 1826354 T4437 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1826360 T4437 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1826361 T4437 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1826362 T4437 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1826362 T4437 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1826363 T4437 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1826364 T4437 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1826365 T4437 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1826366 T4437 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1826366 T4437 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1826369 T4437 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1826370 T4437 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1826370 T4437 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1826371 T4437 oasc.RequestHandler
[...truncated too long message...]
3355/ui_s/collection1, http://127.0.0.1:53338/ui_s/collection1, http://127.0.0.1:53320/ui_s/collection1, http://127.0.0.1:53360/ui_s/collection1, http://127.0.0.1:53345/ui_s/collection1]
[junit4] > at __randomizedtesting.SeedInfo.seed([79D554505CB7677B:F833DA482BE80747]:0)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:337)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:636)
[junit4] > at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
[junit4] > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1140)
[junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:138)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] > Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:53338/ui_s/collection1
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:476)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:268)
[junit4] > ... 46 more
[junit4] > Caused by: java.net.SocketException: Invalid argument
[junit4] > at java.net.SocketInputStream.socketRead0(Native Method)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:152)
[junit4] > at java.net.SocketInputStream.read(SocketInputStream.java:122)
[junit4] > at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
[junit4] > at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
[junit4] > at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
[junit4] > at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
[junit4] > at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
[junit4] > at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
[junit4] > at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
[junit4] > at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
[junit4] > at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
[junit4] > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4] > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:366)
[junit4] > ... 48 more
[junit4] 2> 1923647 T4362 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 120250 T4361 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> 1924290 T4595 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: test params are: codec=Lucene45: {range_facet_l=PostingsFormat(name=Lucene41WithOrds), text=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=TestBloomFilteredLucene41Postings), rnd_b=Lucene41(blocksize=128), intDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), timestamp=PostingsFormat(name=Lucene41WithOrds), id=Lucene41(blocksize=128), a_t=PostingsFormat(name=Lucene41WithOrds), range_facet_sl=Lucene41(blocksize=128), range_facet_si=PostingsFormat(name=Lucene41WithOrds), other_tl1=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=PostingsFormat(name=Lucene41WithOrds), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings)}, docValues:{}, sim=DefaultSimilarity, locale=no_NO_NY, timezone=America/Argentina/Mendoza
[junit4] 2> NOTE: Mac OS X 10.8.5 x86_64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=2,threads=1,free=106621504,total=238419968
[junit4] 2> NOTE: All tests run in this JVM: [TestRemoteStreaming, TestCoreContainer, BlockCacheTest, RegexBoostProcessorTest, SchemaVersionSpecificBehaviorTest, HdfsUnloadDistributedZkTest, TestSolrIndexConfig, QueryEqualityTest, JsonLoaderTest, TestSolrDeletionPolicy2, TestReloadAndDeleteDocs, TermVectorComponentTest, TestCloudManagedSchemaAddField, TestManagedSchema, EchoParamsTest, BasicDistributedZkTest, CollectionsAPIDistributedZkTest, TestSweetSpotSimilarityFactory, TestLFUCache, ParsingFieldUpdateProcessorsTest, SimplePostToolTest, HardAutoCommitTest, ZkSolrClientTest, SolrCoreCheckLockOnStartupTest, TestCloudManagedSchema, TestJmxMonitoredMap, PreAnalyzedUpdateProcessorTest, TestPhraseSuggestions, TestDistributedSearch, ResponseLogComponentTest, XmlUpdateRequestHandlerTest, PolyFieldTest, TestIndexSearcher, SliceStateTest, NotRequiredUniqueKeyTest, DebugComponentTest, DocValuesTest, NoCacheHeaderTest, TestAnalyzedSuggestions, TestRTGBase, TestAtomicUpdateErrorCases, TestSolrQueryParserDefaultOperatorResource, TimeZoneUtilsTest, TestDocSet, TestStressRecovery, MBeansHandlerTest, PreAnalyzedFieldTest, TestSchemaVersionResource, TestPostingsSolrHighlighter, ShardSplitTest, AliasIntegrationTest, TestUtils, TestStressReorder, TestDynamicFieldCollectionResource, BasicDistributedZk2Test, TestSystemIdResolver, TestSolrCoreProperties, HdfsSyncSliceTest, TestDocumentBuilder, TestClassNameShortening, HighlighterTest, SolrIndexSplitterTest, LeaderElectionTest, ZkCLITest, MinimalSchemaTest, LukeRequestHandlerTest, ShardRoutingCustomTest, TestJmxIntegration, ClusterStateUpdateTest, SOLR749Test, QueryParsingTest, CachingDirectoryFactoryTest, FileBasedSpellCheckerTest, ScriptEngineTest, DistanceFunctionTest, OpenCloseCoreStressTest, PluginInfoTest, URLClassifyProcessorTest, TestPHPSerializedResponseWriter, TestCopyFieldCollectionResource, BadCopyFieldTest, CircularListTest, TestValueSourceCache, TestPerFieldSimilarity, SuggesterFSTTest, TestSolrQueryParser, UpdateParamsTest, TestCollationField, HdfsLockFactoryTest, TestCoreDiscovery, TestBM25SimilarityFactory, PeerSyncTest, TestConfig, TestNoOpRegenerator, TestZkChroot, ZkNodePropsTest, SolrXmlInZkTest, TestNumberUtils, TestCharFilters, DistributedSpellCheckComponentTest, PrimitiveFieldTypeTest, BJQParserTest, FieldMutatingUpdateProcessorTest, TestUpdate, DirectUpdateHandlerOptimizeTest, TestFunctionQuery, TestInfoStreamLogging, NumericFieldsTest, TestQueryUtils, DocValuesMultiTest, CSVRequestHandlerTest, TestMergePolicyConfig, OverseerTest, ShardRoutingTest, TestAnalyzeInfixSuggestions, AssignTest, BlockDirectoryTest, TestFieldTypeCollectionResource, TestCodecSupport, TestUniqueKeyFieldResource, TestLRUCache, SearchHandlerTest, DateFieldTest, BadIndexSchemaTest, TestSolrJ, InfoHandlerTest, HdfsChaosMonkeySafeLeaderTest, TestWriterPerf, TestRandomMergePolicy, TestComponentsName, UpdateRequestProcessorFactoryTest, TestLMDirichletSimilarityFactory, TestPseudoReturnFields, TestFieldResource, BadComponentTest, DefaultValueUpdateProcessorTest, TestShardHandlerFactory, TestCSVLoader, TestBinaryField, DateMathParserTest, SolrIndexConfigTest, TestDefaultSearchFieldResource, CoreAdminHandlerTest, SolrTestCaseJ4Test, SystemInfoHandlerTest, TestNRTOpen, TestCollationKeyRangeQueries, FieldAnalysisRequestHandlerTest, SpellingQueryConverterTest, HdfsBasicDistributedZkTest, DistributedQueryElevationComponentTest, TestMultiCoreConfBootstrap, SolrCoreTest, TestStressLucene, DocValuesMissingTest, HdfsBasicDistributedZk2Test, ChaosMonkeySafeLeaderTest]
[junit4] Completed in 121.05s, 1 test, 1 error <<< FAILURES!
[...truncated 505 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:428: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:408: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/build.xml:39: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/build.xml:189: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/solr/common-build.xml:483: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:1263: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-4.x-MacOSX/lucene/common-build.xml:906: There were test failures: 327 suites, 1453 tests, 1 error, 40 ignored (5 assumptions)
Total time: 106 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure