You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/03/11 17:29:01 UTC
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2334 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2334/
All tests passed
Build Log:
[...truncated 27817 lines...]
check-licenses:
[echo] License check under: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr
[licenses] MISSING sha1 checksum file for: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/test-framework/lib/junit4-ant-2.1.1.jar
[licenses] EXPECTED sha1 checksum file : /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/licenses/junit4-ant-2.1.1.jar.sha1
[...truncated 3 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:473: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:70: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:254: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/tools/custom-tasks.xml:62: License check failed. Check the logs.
Total time: 97 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2335 - Still
Failing
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2335/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch
Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:24974/collection1, https://127.0.0.1:56365/collection1]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:24974/collection1, https://127.0.0.1:56365/collection1]
at __randomizedtesting.SeedInfo.seed([808B4AD3C89C5989:16DC4CBBFC339B5]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:387)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1240)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1209)
at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:213)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1617)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:826)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:862)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:876)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:783)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:443)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:835)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:771)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:782)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:359)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: no servers hosting shard:
at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:504)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:203)
at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:302)
... 46 more
Build Log:
[...truncated 10473 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> 1788214 T6475 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 1788215 T6475 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 1788221 T6475 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1394613583565
[junit4] 2> 1788222 T6475 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1788223 T6476 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1788324 T6475 oasc.ZkTestServer.run start zk server on port:10952
[junit4] 2> 1788325 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1788385 T6482 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42eb4a2c name:ZooKeeperConnection Watcher:127.0.0.1:10952 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1788385 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1788386 T6475 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1788392 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1788404 T6484 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57b2a23 name:ZooKeeperConnection Watcher:127.0.0.1:10952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1788404 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1788405 T6475 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1788409 T6475 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1788417 T6475 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1788420 T6475 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1788422 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1788423 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1788434 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1788435 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1788555 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1788556 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1788580 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1788581 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1788584 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1788585 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1788632 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1788632 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1788636 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1788636 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1788639 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1788640 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1788651 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1788652 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1788655 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1788656 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1788659 T6475 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1788659 T6475 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1788671 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1788673 T6486 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6bba68d name:ZooKeeperConnection Watcher:127.0.0.1:10952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1788673 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1789640 T6475 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1789644 T6475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1] of [SSLv2Hello, SSLv3, TLSv1]
[junit4] 2> 1789647 T6475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:27082
[junit4] 2> 1789649 T6475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1789649 T6475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1789650 T6475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020
[junit4] 2> 1789650 T6475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/'
[junit4] 2> 1789675 T6475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/solr.xml
[junit4] 2> 1789728 T6475 oasc.CoreContainer.<init> New CoreContainer 1124393924
[junit4] 2> 1789729 T6475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/]
[junit4] 2> 1789730 T6475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1789730 T6475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1789740 T6475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1789741 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1789741 T6475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1789742 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
[junit4] 2> 1789742 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1789743 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1789743 T6475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1789744 T6475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1789758 T6475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1789759 T6475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1789759 T6475 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1789760 T6475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:10952/solr
[junit4] 2> 1789761 T6475 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1789762 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1789789 T6497 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@31cecbc8 name:ZooKeeperConnection Watcher:127.0.0.1:10952 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1789790 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1789806 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1789808 T6499 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a11b3ae name:ZooKeeperConnection Watcher:127.0.0.1:10952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1789808 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1789811 T6475 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1789820 T6475 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1789823 T6475 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1789825 T6475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27082_
[junit4] 2> 1789833 T6475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27082_
[junit4] 2> 1789836 T6475 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1789838 T6475 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1789842 T6475 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:27082_
[junit4] 2> 1789843 T6475 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1789845 T6475 oasc.Overseer.start Overseer (id=91397395813040132-127.0.0.1:27082_-n_0000000000) starting
[junit4] 2> 1789849 T6475 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1789864 T6501 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1789865 T6475 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1789865 T6501 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 1789867 T6475 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1789868 T6475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1789870 T6500 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1789877 T6502 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 1789877 T6502 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1789878 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1789879 T6502 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1789880 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1789880 T6500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:27082",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:27082_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1789881 T6500 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1789881 T6500 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1789906 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1789908 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1790880 T6502 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1790880 T6502 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/collection1
[junit4] 2> 1790881 T6502 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1790881 T6502 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1790882 T6502 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1790883 T6502 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1790883 T6502 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/collection1/'
[junit4] 2> 1790885 T6502 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/collection1/lib/classes/' to classloader
[junit4] 2> 1790886 T6502 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/collection1/lib/.svn/' to classloader
[junit4] 2> 1790886 T6502 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/collection1/lib/README' to classloader
[junit4] 2> 1790957 T6502 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 1791025 T6502 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1791127 T6502 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1791150 T6502 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1791708 T6502 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1791712 T6502 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1791714 T6502 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1791720 T6502 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1791741 T6502 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1791742 T6502 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1394613584020/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/control/data/
[junit4] 2> 1791742 T6502 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7131359
[junit4] 2> 1791744 T6502 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/control/data
[junit4] 2> 1791745 T6502 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/control/data/index/
[junit4] 2> 1791746 T6502 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1791747 T6502 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/control/data/index
[junit4] 2> 1791748 T6502 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=9, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2507438958993705]
[junit4] 2> 1791752 T6502 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1791753 T6502 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1791756 T6502 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1791756 T6502 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1791757 T6502 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1791757 T6502 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1791757 T6502 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1791758 T6502 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1791758 T6502 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1791759 T6502 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1791759 T6502 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1791760 T6502 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1791760 T6502 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1791761 T6502 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1791761 T6502 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1791762 T6502 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1791762 T6502 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1791763 T6502 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1791772 T6502 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1791777 T6502 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1791777 T6502 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1791778 T6502 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=81.4013671875, floorSegmentMB=0.47265625, forceMergeDeletesPctAllowed=4.468474225898531, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.14864930448201658
[junit4] 2> 1791779 T6502 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1791780 T6502 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1791780 T6502 oass.SolrIndexSearcher.<init> Opening Searcher@9976afa[collection1] main
[junit4] 2> 1791784 T6503 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@9976afa[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1791786 T6502 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1791787 T6475 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1791787 T6475 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1791787 T6506 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:27082 collection:control_collection shard:shard1
[junit4] 2> 1791789 T6506 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1791793 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1791801 T6508 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e4103fa name:ZooKeeperConnection Watcher:127.0.0.1:10952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1791801 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1791806 T6475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1791809 T6506 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1791809 T6475 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1791819 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1791820 T6506 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1791820 T6506 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1513 name=collection1 org.apache.solr.core.SolrCore@4b704bc6 url=https://127.0.0.1:27082/collection1 node=127.0.0.1:27082_ C1513_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:27082, core=collection1, node_name=127.0.0.1:27082_}
[junit4] 2> 1791820 T6506 C1513 P27082 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:27082/collection1/
[junit4] 2> 1791820 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1791821 T6506 C1513 P27082 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1791821 T6506 C1513 P27082 oasc.SyncStrategy.syncToMe https://127.0.0.1:27082/collection1/ has no replicas
[junit4] 2> 1791822 T6506 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:27082/collection1/ shard1
[junit4] 2> 1791822 T6506 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1791825 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1791834 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1791834 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1791854 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1791857 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1791961 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1791961 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1792006 T6506 oasc.ZkController.register We are https://127.0.0.1:27082/collection1/ and leader is https://127.0.0.1:27082/collection1/
[junit4] 2> 1792006 T6506 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:27082
[junit4] 2> 1792006 T6506 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1792006 T6506 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 1792007 T6506 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1792008 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1792009 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1792009 T6506 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1792009 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1792010 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1792011 T6500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:27082",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:27082_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1792014 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1792116 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1792116 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1792852 T6475 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 1792853 T6475 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1792857 T6475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1] of [SSLv2Hello, SSLv3, TLSv1]
[junit4] 2> 1792860 T6475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:64088
[junit4] 2> 1792862 T6475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1792863 T6475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1792864 T6475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153
[junit4] 2> 1792864 T6475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/'
[junit4] 2> 1792902 T6475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/solr.xml
[junit4] 2> 1792981 T6475 oasc.CoreContainer.<init> New CoreContainer 1224955016
[junit4] 2> 1792982 T6475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/]
[junit4] 2> 1792984 T6475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1792984 T6475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1792985 T6475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1792985 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1792986 T6475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1792986 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1792987 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1792988 T6475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1792988 T6475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1793002 T6475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1793003 T6475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1793003 T6475 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1793004 T6475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:10952/solr
[junit4] 2> 1793004 T6475 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1793005 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1793008 T6519 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a0c45eb name:ZooKeeperConnection Watcher:127.0.0.1:10952 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793009 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1793025 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1793028 T6521 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66aa76d4 name:ZooKeeperConnection Watcher:127.0.0.1:10952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1793028 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1793039 T6475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1794043 T6475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:64088_
[junit4] 2> 1794047 T6475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:64088_
[junit4] 2> 1794051 T6499 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1794051 T6521 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1794051 T6508 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1794061 T6522 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1794061 T6522 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1794063 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1794063 T6522 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1794063 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1794063 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1794065 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1794065 T6500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:64088",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:64088_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1794066 T6500 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 1794066 T6500 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1794069 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1794070 T6521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1794070 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1794070 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1795064 T6522 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1795064 T6522 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/collection1
[junit4] 2> 1795064 T6522 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1795065 T6522 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1795065 T6522 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1795066 T6522 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1795067 T6522 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/collection1/'
[junit4] 2> 1795069 T6522 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/collection1/lib/.svn/' to classloader
[junit4] 2> 1795069 T6522 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/collection1/lib/README' to classloader
[junit4] 2> 1795070 T6522 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/collection1/lib/classes/' to classloader
[junit4] 2> 1795141 T6522 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 1795218 T6522 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1795319 T6522 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1795341 T6522 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1795897 T6522 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1795901 T6522 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1795903 T6522 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1795909 T6522 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1795930 T6522 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1795931 T6522 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1394613587153/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty1/
[junit4] 2> 1795931 T6522 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7131359
[junit4] 2> 1795932 T6522 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty1
[junit4] 2> 1795933 T6522 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty1/index/
[junit4] 2> 1795934 T6522 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1795936 T6522 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty1/index
[junit4] 2> 1795936 T6522 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=9, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2507438958993705]
[junit4] 2> 1795945 T6522 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1795946 T6522 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1795950 T6522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1795950 T6522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1795951 T6522 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1795951 T6522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1795952 T6522 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1795952 T6522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1795952 T6522 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1795953 T6522 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1795953 T6522 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1795954 T6522 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1795955 T6522 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1795955 T6522 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1795956 T6522 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1795956 T6522 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1795957 T6522 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1795958 T6522 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1795967 T6522 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1795971 T6522 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1795972 T6522 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1795973 T6522 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=81.4013671875, floorSegmentMB=0.47265625, forceMergeDeletesPctAllowed=4.468474225898531, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.14864930448201658
[junit4] 2> 1795974 T6522 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1795975 T6522 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1795975 T6522 oass.SolrIndexSearcher.<init> Opening Searcher@15fbc6c7[collection1] main
[junit4] 2> 1795981 T6523 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@15fbc6c7[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1795984 T6522 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1795985 T6475 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1795985 T6475 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1795985 T6526 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:64088 collection:collection1 shard:shard1
[junit4] 2> 1795987 T6526 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1796006 T6526 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1796008 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1796009 T6526 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1796009 T6526 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1514 name=collection1 org.apache.solr.core.SolrCore@598c18a2 url=https://127.0.0.1:64088/collection1 node=127.0.0.1:64088_ C1514_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:64088, core=collection1, node_name=127.0.0.1:64088_}
[junit4] 2> 1796009 T6526 C1514 P64088 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:64088/collection1/
[junit4] 2> 1796010 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1796010 T6526 C1514 P64088 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1796010 T6526 C1514 P64088 oasc.SyncStrategy.syncToMe https://127.0.0.1:64088/collection1/ has no replicas
[junit4] 2> 1796010 T6526 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:64088/collection1/ shard1
[junit4] 2> 1796011 T6526 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1796013 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1796014 T6521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1796014 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1796014 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1796018 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1796021 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1796124 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1796124 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1796124 T6521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1796171 T6526 oasc.ZkController.register We are https://127.0.0.1:64088/collection1/ and leader is https://127.0.0.1:64088/collection1/
[junit4] 2> 1796171 T6526 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:64088
[junit4] 2> 1796171 T6526 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1796172 T6526 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1796172 T6526 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1796173 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1796174 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1796174 T6526 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1796174 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1796176 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1796176 T6500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:64088",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:64088_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1796179 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1796281 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1796281 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1796281 T6521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1797031 T6475 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 1797033 T6475 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1797037 T6475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1] of [SSLv2Hello, SSLv3, TLSv1]
[junit4] 2> 1797039 T6475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:20809
[junit4] 2> 1797042 T6475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1797043 T6475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1797043 T6475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331
[junit4] 2> 1797044 T6475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/'
[junit4] 2> 1797082 T6475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/solr.xml
[junit4] 2> 1797161 T6475 oasc.CoreContainer.<init> New CoreContainer 1895906646
[junit4] 2> 1797162 T6475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/]
[junit4] 2> 1797164 T6475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1797164 T6475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1797165 T6475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1797165 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1797166 T6475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1797166 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1797167 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1797167 T6475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1797168 T6475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1797182 T6475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1797182 T6475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1797183 T6475 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1797184 T6475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:10952/solr
[junit4] 2> 1797184 T6475 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1797185 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1797189 T6537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17242cbe name:ZooKeeperConnection Watcher:127.0.0.1:10952 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1797190 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1797193 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1797195 T6539 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@773f4fbe name:ZooKeeperConnection Watcher:127.0.0.1:10952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1797195 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1797210 T6475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1798213 T6475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:20809_
[junit4] 2> 1798216 T6475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:20809_
[junit4] 2> 1798220 T6508 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1798221 T6521 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1798221 T6539 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1798221 T6499 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1798240 T6540 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1798240 T6540 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1798242 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1798242 T6540 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1798242 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1798243 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1798244 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1798245 T6500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:20809",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:20809_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1798245 T6500 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1798245 T6500 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1798249 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1798255 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1798256 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1798255 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1798255 T6521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1799243 T6540 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1799243 T6540 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/collection1
[junit4] 2> 1799244 T6540 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1799244 T6540 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1799245 T6540 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1799246 T6540 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 1799246 T6540 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/collection1/'
[junit4] 2> 1799247 T6540 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/collection1/lib/README' to classloader
[junit4] 2> 1799248 T6540 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/collection1/lib/.svn/' to classloader
[junit4] 2> 1799249 T6540 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/collection1/lib/classes/' to classloader
[junit4] 2> 1799320 T6540 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 1799389 T6540 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1799490 T6540 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1799513 T6540 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1800079 T6540 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1800083 T6540 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1800086 T6540 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1800091 T6540 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1800110 T6540 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1800110 T6540 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1394613591331/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty2/
[junit4] 2> 1800110 T6540 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7131359
[junit4] 2> 1800112 T6540 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty2
[junit4] 2> 1800113 T6540 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty2/index/
[junit4] 2> 1800114 T6540 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1800115 T6540 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty2/index
[junit4] 2> 1800116 T6540 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=9, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2507438958993705]
[junit4] 2> 1800121 T6540 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1800121 T6540 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1800124 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1800125 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1800125 T6540 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1800126 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1800126 T6540 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1800126 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1800127 T6540 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1800127 T6540 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1800128 T6540 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1800128 T6540 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1800129 T6540 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1800129 T6540 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1800130 T6540 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1800130 T6540 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1800131 T6540 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1800131 T6540 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1800141 T6540 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1800146 T6540 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1800146 T6540 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1800147 T6540 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=81.4013671875, floorSegmentMB=0.47265625, forceMergeDeletesPctAllowed=4.468474225898531, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.14864930448201658
[junit4] 2> 1800149 T6540 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1394613583564/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1800149 T6540 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1800150 T6540 oass.SolrIndexSearcher.<init> Opening Searcher@1a64b4e2[collection1] main
[junit4] 2> 1800153 T6541 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a64b4e2[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1800155 T6540 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1800156 T6475 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1800157 T6475 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1800156 T6544 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:20809 collection:collection1 shard:shard2
[junit4] 2> 1800158 T6544 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1800174 T6544 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1800176 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1800177 T6544 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1800177 T6544 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1515 name=collection1 org.apache.solr.core.SolrCore@7c49013c url=https://127.0.0.1:20809/collection1 node=127.0.0.1:20809_ C1515_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:20809, core=collection1, node_name=127.0.0.1:20809_}
[junit4] 2> 1800177 T6544 C1515 P20809 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:20809/collection1/
[junit4] 2> 1800177 T6544 C1515 P20809 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1800177 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1800177 T6544 C1515 P20809 oasc.SyncStrategy.syncToMe https://127.0.0.1:20809/collection1/ has no replicas
[junit4] 2> 1800178 T6544 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:20809/collection1/ shard2
[junit4] 2> 1800178 T6544 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1800187 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1800197 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800197 T6521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800197 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800197 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800202 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1800210 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1800315 T6521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800315 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800315 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800315 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800353 T6544 oasc.ZkController.register We are https://127.0.0.1:20809/collection1/ and leader is https://127.0.0.1:20809/collection1/
[junit4] 2> 1800353 T6544 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:20809
[junit4] 2> 1800353 T6544 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1800353 T6544 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 1800354 T6544 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1800355 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1800355 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1800355 T6544 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1800356 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1800357 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1800357 T6500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:20809",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:20809_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1800399 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1800501 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800501 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800501 T6499 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800501 T6521 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1800860 T6475 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 1800861 T6475 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1800865 T6475 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1] of [SSLv2Hello, SSLv3, TLSv1]
[junit4] 2> 1800866 T6475 oejs.AbstractConnector.doStart Started SslSocketConnector@127.0.0.1:24974
[junit4] 2> 1800868 T6475 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1800869 T6475 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1800869 T6475 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1394613595502
[junit4] 2> 1800870 T6475 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1394613595502/'
[junit4] 2> 1800895 T6475 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1394613595502/solr.xml
[junit4] 2> 1800948 T6475 oasc.CoreContainer.<init> New CoreContainer 1654340402
[junit4] 2> 1800949 T6475 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1394613595502/]
[junit4] 2> 1800950 T6475 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1800951 T6475 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 1800951 T6475 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1800951 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1800952 T6475 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1800952 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1800952 T6475 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1800953 T6475 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1800953 T6475 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1800962 T6475 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1800963 T6475 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1800963 T6475 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1800964 T6475 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:10952/solr
[junit4] 2> 1800964 T6475 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1800965 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1800967 T6555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5b3f91a1 name:ZooKeeperConnection Watcher:127.0.0.1:10952 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1800968 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1800979 T6475 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1800980 T6557 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d9e6e23 name:ZooKeeperConnection Watcher:127.0.0.1:10952/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1800981 T6475 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1800993 T6475 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1801996 T6475 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:24974_
[junit4] 2> 1801998 T6475 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:24974_
[junit4] 2> 1802003 T6499 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1802003 T6557 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1802003 T6521 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1802003 T6539 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1802003 T6508 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1802023 T6558 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 1802023 T6558 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1802025 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1802025 T6558 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1802025 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1802025 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1802026 T6500 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1802027 T6500 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:24974",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:24974_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1802027 T6500 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1802027 T6500 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1802030 T6499 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1802037 T6539 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1802037 T6508 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live
[...truncated too long message...]
_name":"127.0.0.1:27082_",
[junit4] 1> "base_url":"https://127.0.0.1:27082"}
[junit4] 1> /solr/overseer_elect (2)
[junit4] 1> /solr/overseer_elect/election (4)
[junit4] 1> /solr/overseer_elect/election/91397395813040132-127.0.0.1:27082_-n_0000000000 (0)
[junit4] 1> /solr/overseer_elect/election/91397395813040141-127.0.0.1:37699_-n_0000000004 (0)
[junit4] 1> /solr/overseer_elect/election/91397395813040139-127.0.0.1:24974_-n_0000000003 (0)
[junit4] 1> /solr/overseer_elect/election/91397395813040143-127.0.0.1:56365_-n_0000000005 (0)
[junit4] 1> /solr/overseer_elect/leader (0)
[junit4] 1> DATA:
[junit4] 1> {"id":"91397395813040132-127.0.0.1:27082_-n_0000000000"}
[junit4] 1> /zookeeper (1)
[junit4] 1> DATA:
[junit4] 1>
[junit4] 1>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch -Dtests.seed=808B4AD3C89C5989 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_OM -Dtests.timezone=Mexico/General -Dtests.file.encoding=US-ASCII
[junit4] ERROR 606s J1 | ChaosMonkeyNothingIsSafeTest.testDistribSearch <<<
[junit4] > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:24974/collection1, https://127.0.0.1:56365/collection1]
[junit4] > at __randomizedtesting.SeedInfo.seed([808B4AD3C89C5989:16DC4CBBFC339B5]:0)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:387)
[junit4] > at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:635)
[junit4] > at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91)
[junit4] > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1240)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1209)
[junit4] > at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:213)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:870)
[junit4] > at java.lang.Thread.run(Thread.java:679)
[junit4] > Caused by: org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: no servers hosting shard:
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:504)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:203)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
[junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:302)
[junit4] > ... 46 more
[junit4] 2> 2394405 T6475 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 606196 T6474 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Lucene46: {id=FST41, a_t=PostingsFormat(name=TestBloomFilteredLucene41Postings), _version_=PostingsFormat(name=TestBloomFilteredLucene41Postings), rnd_b=FST41, a_si=MockVariableIntBlock(baseBlockSize=19)}, docValues:{}, sim=DefaultSimilarity, locale=ar_OM, timezone=Mexico/General
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=140520296,total=453836800
[junit4] 2> NOTE: All tests run in this JVM: [TestRandomMergePolicy, PluginInfoTest, InfoHandlerTest, HdfsCollectionsAPIDistributedZkTest, TestCodecSupport, TestComponentsName, TestWordDelimiterFilterFactory, ChaosMonkeySafeLeaderTest, FileUtilsTest, TestFieldTypeResource, TestIBSimilarityFactory, TestDistributedGrouping, TestUtils, SpellingQueryConverterTest, TestFieldCollectionResource, TestAddFieldRealTimeGet, TestNumberUtils, RequiredFieldsTest, DocValuesMissingTest, TestDefaultSimilarityFactory, CursorPagingTest, TestBinaryResponseWriter, TestCollationField, TestSolrQueryParserResource, TestPartialUpdateDeduplication, SolrCoreCheckLockOnStartupTest, PingRequestHandlerTest, CachingDirectoryFactoryTest, BadIndexSchemaTest, OpenCloseCoreStressTest, DistanceFunctionTest, TestIndexingPerformance, TestQuerySenderListener, TestAnalyzedSuggestions, SolrTestCaseJ4Test, DirectSolrSpellCheckerTest, HighlighterTest, DocumentAnalysisRequestHandlerTest, PreAnalyzedFieldTest, RequestHandlersTest, CoreContainerCoreInitFailuresTest, DirectSolrConnectionTest, TestSolrQueryParserDefaultOperatorResource, SpellCheckComponentTest, TestHashPartitioner, AlternateDirectoryTest, DebugComponentTest, TestFieldResource, UniqFieldsUpdateProcessorFactoryTest, TestStressRecovery, TestQuerySenderNoQuery, TestHighlightDedupGrouping, BlockDirectoryTest, CurrencyFieldOpenExchangeTest, TestOmitPositions, StatsComponentTest, DefaultValueUpdateProcessorTest, TestCopyFieldCollectionResource, TestFastOutputStream, CurrencyFieldXmlFileTest, TestZkChroot, ConnectionManagerTest, TestDFRSimilarityFactory, HdfsWriteToMultipleCollectionsTest, ZkNodePropsTest, OutputWriterTest, BadCopyFieldTest, ConvertedLegacyTest, TestDistributedMissingSort, CoreAdminCreateDiscoverTest, TestFiltering, BasicZkTest, TestSolrCoreProperties, HdfsBasicDistributedZk2Test, TestQueryTypes, TestReload, TestImplicitCoreProperties, TimeZoneUtilsTest, CollectionsAPIDistributedZkTest, TestManagedSchemaFieldResource, StandardRequestHandlerTest, TestSchemaNameResource, TestArbitraryIndexDir, TestFieldTypeCollectionResource, TestIndexSearcher, TestSolrDeletionPolicy2, TestSchemaSimilarityResource, TestShardHandlerFactory, DistribCursorPagingTest, HdfsUnloadDistributedZkTest, TestMultiCoreConfBootstrap, BasicDistributedZk2Test, LeaderElectionIntegrationTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, ZkSolrClientTest, TestRandomDVFaceting, ShardRoutingCustomTest, TestDistributedSearch, TestRecovery, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, ZkControllerTest, TestStressReorder, HardAutoCommitTest, TestRangeQuery, SimpleFacetsTest, TestSolr4Spatial, SolrCoreTest, TestGroupingSearch, SolrCmdDistributorTest, PeerSyncTest, TestSort, TestLazyCores, DirectUpdateHandlerTest, TestBadConfig, OverseerCollectionProcessorTest, ShowFileRequestHandlerTest, SimplePostToolTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTSTTest, SuggesterTest, TestStressLucene, SuggesterWFSTTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, SortByFunctionTest, DocValuesMultiTest, TestSolrDeletionPolicy1, XsltUpdateRequestHandlerTest, CacheHeaderTest, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestQueryUtils, TestWriterPerf, FileBasedSpellCheckerTest, TermsComponentTest, XmlUpdateRequestHandlerTest, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, RegexBoostProcessorTest, SolrPluginUtilsTest, TestJmxIntegration, JSONWriterTest, TestCSVResponseWriter, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, SearchHandlerTest, HighlighterConfigTest, UpdateParamsTest, ResponseLogComponentTest, SolrIndexConfigTest, TestMergePolicyConfig, MultiTermTest, TestDocSet, TestBinaryField, TestSearchPerf, NumericFieldsTest, MinimalSchemaTest, TestConfig, TestFuzzyAnalyzedSuggestions, NotRequiredUniqueKeyTest, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, SynonymTokenizerTest, TestXIncludeConfig, EchoParamsTest, TestSweetSpotSimilarityFactory, TestLMDirichletSimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, TestJmxMonitoredMap, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest]
[junit4] Completed on J1 in 606.48s, 1 test, 1 error <<< FAILURES!
[...truncated 231 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:473: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:453: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:45: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:490: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1279: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:911: There were test failures: 369 suites, 1581 tests, 1 error, 37 ignored (16 assumptions)
Total time: 74 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure