You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2014/02/05 20:37:32 UTC
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2296 - Still
Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2296/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch
Error Message:
document count mismatch. control=18 sum(shards)=17 cloudClient=17
Stack Trace:
java.lang.AssertionError: document count mismatch. control=18 sum(shards)=17 cloudClient=17
at __randomizedtesting.SeedInfo.seed([FE94D66684AF61FD:7F72587EF3F001C1]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1215)
at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:208)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:867)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
Build Log:
[...truncated 10355 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> 1748936 T5311 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /l_/og
[junit4] 2> 1748942 T5311 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1391628382640
[junit4] 2> 1748944 T5311 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1748945 T5312 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1749046 T5311 oasc.ZkTestServer.run start zk server on port:54739
[junit4] 2> 1749047 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1749053 T5318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59075168 name:ZooKeeperConnection Watcher:127.0.0.1:54739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1749054 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1749054 T5311 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1749060 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1749074 T5320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@207ca082 name:ZooKeeperConnection Watcher:127.0.0.1:54739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1749074 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1749075 T5311 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1749079 T5311 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1749088 T5311 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1749092 T5311 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1749095 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1749095 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1749105 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1749106 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1749211 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1749212 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1749215 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1749216 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1749219 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1749220 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1749227 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1749228 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1749231 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1749231 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 1749235 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1749235 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1749238 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1749239 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1749242 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1749243 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1749246 T5311 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1749246 T5311 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1749549 T5311 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1749555 T5311 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:29072
[junit4] 2> 1749555 T5311 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1749556 T5311 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1749557 T5311 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954
[junit4] 2> 1749557 T5311 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954/'
[junit4] 2> 1749597 T5311 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954/solr.xml
[junit4] 2> 1749678 T5311 oasc.CoreContainer.<init> New CoreContainer 676555059
[junit4] 2> 1749679 T5311 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954/]
[junit4] 2> 1749681 T5311 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1749681 T5311 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1749682 T5311 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1749682 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1749683 T5311 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1749683 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1749684 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1749685 T5311 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1749685 T5311 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1749695 T5311 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1749695 T5311 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1749696 T5311 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1749697 T5311 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54739/solr
[junit4] 2> 1749697 T5311 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1749698 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1749701 T5331 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c1fa7f3 name:ZooKeeperConnection Watcher:127.0.0.1:54739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1749702 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1749705 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1749714 T5333 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e18a85e name:ZooKeeperConnection Watcher:127.0.0.1:54739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1749715 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1749717 T5311 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1749726 T5311 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1749729 T5311 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1749737 T5311 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:29072_l_%2Fog
[junit4] 2> 1749738 T5311 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:29072_l_%2Fog
[junit4] 2> 1749741 T5311 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 1749744 T5311 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1749759 T5311 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:29072_l_%2Fog
[junit4] 2> 1749760 T5311 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1749762 T5311 oasc.Overseer.start Overseer (id=91201757685219331-127.0.0.1:29072_l_%2Fog-n_0000000000) starting
[junit4] 2> 1749775 T5311 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1749785 T5335 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1749785 T5311 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1749785 T5335 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes
[junit4] 2> 1749785 T5334 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1749788 T5311 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1749790 T5311 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1749798 T5336 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1749799 T5336 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1749800 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1749800 T5336 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1749802 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1749802 T5334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:29072/l_/og",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:29072_l_%2Fog",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1749803 T5334 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1749803 T5334 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1749811 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1749812 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1750802 T5336 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1750802 T5336 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954/collection1
[junit4] 2> 1750803 T5336 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1750804 T5336 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1750804 T5336 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1750805 T5336 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954/collection1/'
[junit4] 2> 1750807 T5336 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954/collection1/lib/classes/' to classloader
[junit4] 2> 1750808 T5336 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954/collection1/lib/README' to classloader
[junit4] 2> 1750874 T5336 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 1750953 T5336 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1751054 T5336 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1751064 T5336 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1751672 T5336 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1751676 T5336 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1751679 T5336 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1751684 T5336 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1751704 T5336 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1751705 T5336 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1391628382954/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/control/data/
[junit4] 2> 1751705 T5336 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@618d2c82
[junit4] 2> 1751707 T5336 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/control/data
[junit4] 2> 1751708 T5336 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/control/data/index/
[junit4] 2> 1751709 T5336 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1751710 T5336 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/control/data/index
[junit4] 2> 1751711 T5336 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1751715 T5336 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1751716 T5336 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1751719 T5336 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1751719 T5336 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1751720 T5336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1751720 T5336 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1751721 T5336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1751721 T5336 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1751722 T5336 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1751722 T5336 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1751722 T5336 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1751723 T5336 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1751724 T5336 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1751724 T5336 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1751724 T5336 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1751725 T5336 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1751725 T5336 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1751726 T5336 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1751736 T5336 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1751740 T5336 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1751740 T5336 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1751741 T5336 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=35, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1498080075045065]
[junit4] 2> 1751743 T5336 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1751743 T5336 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1751744 T5336 oass.SolrIndexSearcher.<init> Opening Searcher@40459046 main
[junit4] 2> 1751746 T5337 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@40459046 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1751748 T5336 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1751749 T5311 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1751750 T5311 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1751749 T5340 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:29072/l_/og collection:control_collection shard:shard1
[junit4] 2> 1751751 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1751752 T5340 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1751778 T5342 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ba1335e name:ZooKeeperConnection Watcher:127.0.0.1:54739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1751779 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1751792 T5311 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1751804 T5311 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1751805 T5340 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1751816 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1751816 T5340 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1751817 T5340 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1676 name=collection1 org.apache.solr.core.SolrCore@2149a168 url=http://127.0.0.1:29072/l_/og/collection1 node=127.0.0.1:29072_l_%2Fog C1676_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:29072/l_/og, core=collection1, node_name=127.0.0.1:29072_l_%2Fog}
[junit4] 2> 1751817 T5340 C1676 P29072 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:29072/l_/og/collection1/
[junit4] 2> 1751817 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1751817 T5340 C1676 P29072 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1751818 T5340 C1676 P29072 oasc.SyncStrategy.syncToMe http://127.0.0.1:29072/l_/og/collection1/ has no replicas
[junit4] 2> 1751818 T5340 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:29072/l_/og/collection1/ shard1
[junit4] 2> 1751819 T5340 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1751820 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1751829 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1751829 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1751840 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1751841 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1751851 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1751955 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1751956 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1751993 T5340 oasc.ZkController.register We are http://127.0.0.1:29072/l_/og/collection1/ and leader is http://127.0.0.1:29072/l_/og/collection1/
[junit4] 2> 1751993 T5340 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:29072/l_/og
[junit4] 2> 1751993 T5340 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1751993 T5340 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1751994 T5340 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1751995 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1751995 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1751996 T5340 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1751996 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1751997 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1751998 T5334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:29072/l_/og",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:29072_l_%2Fog",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1752000 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1752085 T5311 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 1752086 T5311 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1752088 T5311 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:36106
[junit4] 2> 1752089 T5311 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1752090 T5311 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1752091 T5311 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502
[junit4] 2> 1752091 T5311 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502/'
[junit4] 2> 1752108 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1752108 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1752131 T5311 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502/solr.xml
[junit4] 2> 1752212 T5311 oasc.CoreContainer.<init> New CoreContainer 419928131
[junit4] 2> 1752213 T5311 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502/]
[junit4] 2> 1752215 T5311 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1752216 T5311 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1752216 T5311 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1752217 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1752217 T5311 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1752218 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1752218 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1752219 T5311 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1752219 T5311 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1752229 T5311 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1752229 T5311 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1752230 T5311 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1752230 T5311 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54739/solr
[junit4] 2> 1752231 T5311 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1752232 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1752239 T5353 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@419d638b name:ZooKeeperConnection Watcher:127.0.0.1:54739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1752240 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1752243 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1752245 T5355 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cfafa47 name:ZooKeeperConnection Watcher:127.0.0.1:54739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1752245 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1752255 T5311 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1753259 T5311 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36106_l_%2Fog
[junit4] 2> 1753287 T5311 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36106_l_%2Fog
[junit4] 2> 1753291 T5342 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1753291 T5355 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1753291 T5333 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1753304 T5356 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1753305 T5356 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1753306 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1753306 T5356 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1753307 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1753307 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1753309 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1753309 T5334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:36106/l_/og",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:36106_l_%2Fog",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1753310 T5334 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 1753310 T5334 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1753313 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1753416 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1753416 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1753416 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1754308 T5356 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1754308 T5356 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502/collection1
[junit4] 2> 1754309 T5356 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1754309 T5356 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1754310 T5356 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1754311 T5356 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502/collection1/'
[junit4] 2> 1754312 T5356 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502/collection1/lib/README' to classloader
[junit4] 2> 1754313 T5356 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502/collection1/lib/classes/' to classloader
[junit4] 2> 1754369 T5356 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 1754417 T5356 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1754518 T5356 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1754525 T5356 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1754935 T5356 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1754939 T5356 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1754940 T5356 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1754944 T5356 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1754962 T5356 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1754962 T5356 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1391628385502/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty1/
[junit4] 2> 1754962 T5356 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@618d2c82
[junit4] 2> 1754963 T5356 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty1
[junit4] 2> 1754964 T5356 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty1/index/
[junit4] 2> 1754965 T5356 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1754965 T5356 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty1/index
[junit4] 2> 1754966 T5356 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1754969 T5356 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1754969 T5356 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1754972 T5356 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1754972 T5356 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1754973 T5356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1754973 T5356 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1754973 T5356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1754973 T5356 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1754974 T5356 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1754974 T5356 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1754974 T5356 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1754975 T5356 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1754975 T5356 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1754975 T5356 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1754976 T5356 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1754976 T5356 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1754977 T5356 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1754977 T5356 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1754984 T5356 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1754987 T5356 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1754987 T5356 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1754988 T5356 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=35, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1498080075045065]
[junit4] 2> 1754989 T5356 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1754989 T5356 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1754990 T5356 oass.SolrIndexSearcher.<init> Opening Searcher@1e48881a main
[junit4] 2> 1754993 T5357 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e48881a main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1754994 T5356 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1754995 T5311 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1754995 T5311 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1754995 T5360 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36106/l_/og collection:collection1 shard:shard1
[junit4] 2> 1754997 T5360 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1755022 T5360 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1755024 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1755024 T5360 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1755025 T5360 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1755024 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C1677 name=collection1 org.apache.solr.core.SolrCore@52d09d86 url=http://127.0.0.1:36106/l_/og/collection1 node=127.0.0.1:36106_l_%2Fog C1677_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:36106/l_/og, core=collection1, node_name=127.0.0.1:36106_l_%2Fog}
[junit4] 2> 1755025 T5360 C1677 P36106 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36106/l_/og/collection1/
[junit4] 2> 1755025 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1755025 T5360 C1677 P36106 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1755026 T5360 C1677 P36106 oasc.SyncStrategy.syncToMe http://127.0.0.1:36106/l_/og/collection1/ has no replicas
[junit4] 2> 1755026 T5360 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36106/l_/og/collection1/ shard1
[junit4] 2> 1755026 T5360 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1755027 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1755039 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1755040 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755040 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755040 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755052 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1755055 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1755158 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755158 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755158 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755193 T5311 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 1755194 T5311 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1755197 T5311 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:21615
[junit4] 2> 1755198 T5311 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1755198 T5311 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1755199 T5311 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693
[junit4] 2> 1755199 T5311 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693/'
[junit4] 2> 1755205 T5360 oasc.ZkController.register We are http://127.0.0.1:36106/l_/og/collection1/ and leader is http://127.0.0.1:36106/l_/og/collection1/
[junit4] 2> 1755205 T5360 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36106/l_/og
[junit4] 2> 1755205 T5360 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1755206 T5360 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1755206 T5360 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1755207 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1755208 T5360 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1755208 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1755208 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1755208 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1755209 T5334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:36106/l_/og",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:36106_l_%2Fog",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1755213 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1755228 T5311 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693/solr.xml
[junit4] 2> 1755286 T5311 oasc.CoreContainer.<init> New CoreContainer 853626761
[junit4] 2> 1755287 T5311 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693/]
[junit4] 2> 1755288 T5311 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1755289 T5311 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1755289 T5311 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1755289 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1755290 T5311 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1755290 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1755290 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1755291 T5311 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1755291 T5311 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1755297 T5311 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1755298 T5311 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1755298 T5311 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1755299 T5311 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54739/solr
[junit4] 2> 1755299 T5311 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1755300 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1755302 T5371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@662cc577 name:ZooKeeperConnection Watcher:127.0.0.1:54739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1755303 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1755305 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1755306 T5373 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2555ba8a name:ZooKeeperConnection Watcher:127.0.0.1:54739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1755307 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1755316 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755317 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755317 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1755327 T5311 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1756330 T5311 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:21615_l_%2Fog
[junit4] 2> 1756332 T5311 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:21615_l_%2Fog
[junit4] 2> 1756335 T5342 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1756335 T5355 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1756335 T5373 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1756335 T5333 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1756351 T5374 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1756352 T5374 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1756394 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1756394 T5374 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1756394 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1756395 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1756396 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1756397 T5334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:21615/l_/og",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:21615_l_%2Fog",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1756397 T5334 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1756397 T5334 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1756400 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1756502 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1756502 T5373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1756502 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1756502 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1757396 T5374 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1757396 T5374 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693/collection1
[junit4] 2> 1757397 T5374 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1757398 T5374 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1757398 T5374 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1757399 T5374 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693/collection1/'
[junit4] 2> 1757401 T5374 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693/collection1/lib/README' to classloader
[junit4] 2> 1757402 T5374 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693/collection1/lib/classes/' to classloader
[junit4] 2> 1757467 T5374 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 1757538 T5374 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1757640 T5374 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1757651 T5374 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1758250 T5374 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1758254 T5374 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1758257 T5374 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1758262 T5374 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1758282 T5374 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1758282 T5374 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1391628388693/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty2/
[junit4] 2> 1758283 T5374 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@618d2c82
[junit4] 2> 1758284 T5374 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty2
[junit4] 2> 1758285 T5374 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty2/index/
[junit4] 2> 1758286 T5374 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1758287 T5374 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty2/index
[junit4] 2> 1758288 T5374 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=39, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 1758292 T5374 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1758293 T5374 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1758304 T5374 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1758305 T5374 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1758306 T5374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1758306 T5374 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1758306 T5374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1758307 T5374 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1758307 T5374 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1758308 T5374 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1758308 T5374 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1758309 T5374 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1758310 T5374 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1758310 T5374 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1758310 T5374 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1758311 T5374 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1758312 T5374 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1758312 T5374 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 1758322 T5374 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1758327 T5374 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1758327 T5374 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1758328 T5374 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=35, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1498080075045065]
[junit4] 2> 1758330 T5374 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1758330 T5374 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1758331 T5374 oass.SolrIndexSearcher.<init> Opening Searcher@13730a95 main
[junit4] 2> 1758335 T5375 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@13730a95 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1758337 T5374 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1758338 T5311 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 1758339 T5311 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1758338 T5378 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:21615/l_/og collection:collection1 shard:shard2
[junit4] 2> 1758340 T5378 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1758355 T5378 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1758358 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758358 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758358 T5378 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1758358 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758359 T5378 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1678 name=collection1 org.apache.solr.core.SolrCore@1bc7012a url=http://127.0.0.1:21615/l_/og/collection1 node=127.0.0.1:21615_l_%2Fog C1678_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:21615/l_/og, core=collection1, node_name=127.0.0.1:21615_l_%2Fog}
[junit4] 2> 1758359 T5378 C1678 P21615 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:21615/l_/og/collection1/
[junit4] 2> 1758360 T5378 C1678 P21615 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1758360 T5378 C1678 P21615 oasc.SyncStrategy.syncToMe http://127.0.0.1:21615/l_/og/collection1/ has no replicas
[junit4] 2> 1758360 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1758360 T5378 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:21615/l_/og/collection1/ shard2
[junit4] 2> 1758361 T5378 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1758369 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758370 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758370 T5373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758370 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758370 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758385 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1758388 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758492 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758492 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758492 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758492 T5373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758537 T5378 oasc.ZkController.register We are http://127.0.0.1:21615/l_/og/collection1/ and leader is http://127.0.0.1:21615/l_/og/collection1/
[junit4] 2> 1758537 T5378 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:21615/l_/og
[junit4] 2> 1758537 T5378 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1758538 T5378 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1758538 T5378 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1758540 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758540 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758540 T5378 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1758540 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758541 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1758542 T5334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:21615/l_/og",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:21615_l_%2Fog",
[junit4] 2> "shard":"shard2",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1758545 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1758644 T5311 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 1758645 T5311 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1758647 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758648 T5311 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:22538
[junit4] 2> 1758648 T5373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758649 T5311 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1758648 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758647 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1758650 T5311 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1758651 T5311 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391628392036
[junit4] 2> 1758652 T5311 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391628392036/'
[junit4] 2> 1758695 T5311 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391628392036/solr.xml
[junit4] 2> 1758778 T5311 oasc.CoreContainer.<init> New CoreContainer 1466333063
[junit4] 2> 1758779 T5311 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391628392036/]
[junit4] 2> 1758781 T5311 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1758782 T5311 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1758782 T5311 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1758783 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1758783 T5311 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1758784 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1758785 T5311 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1758785 T5311 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1758786 T5311 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1758796 T5311 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1758797 T5311 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1758798 T5311 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 1758798 T5311 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:54739/solr
[junit4] 2> 1758799 T5311 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1758800 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1758803 T5389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8316ce5 name:ZooKeeperConnection Watcher:127.0.0.1:54739 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1758804 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1758807 T5311 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1758808 T5391 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8dd50f4 name:ZooKeeperConnection Watcher:127.0.0.1:54739/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1758809 T5311 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1758819 T5311 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1759823 T5311 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22538_l_%2Fog
[junit4] 2> 1759825 T5311 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22538_l_%2Fog
[junit4] 2> 1759828 T5342 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1759828 T5355 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1759828 T5333 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1759828 T5391 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1759828 T5373 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1759848 T5392 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1759848 T5392 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1759849 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1759850 T5392 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1759850 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1759850 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1759851 T5334 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1759852 T5334 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:22538/l_/og",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:22538_l_%2Fog",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1759852 T5334 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1759852 T5334 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1759856 T5333 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 1759980 T5391 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1759980 T5355 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1759980 T5373 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1759980 T5333 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1759980 T5342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1760851 T5392 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1760851 T5392 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391628392036/collection1
[junit4] 2> 1760852 T5392 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1760852 T5392 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1760853 T5392 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1760854 T5392 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1391628392036/collection1/'
[...truncated too long message...]
ungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=13,cumulative_deletesById=5,cumulative_deletesByQuery=1,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=1}
[junit4] 2> 2057989 T5470 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 2057990 T5470 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 2057990 T5470 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 2057991 T5470 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 2057992 T5470 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 2057992 T5470 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty7 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty7;done=false>>]
[junit4] 2> 2057993 T5470 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty7
[junit4] 2> 2057993 T5470 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty7/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty7/index;done=false>>]
[junit4] 2> 2057993 T5470 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1391628382640/jetty7/index
[junit4] 2> 2057994 T5470 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2057994 T5470 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 2057995 T5470 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 2057995 T5470 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:256)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:253)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:92)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:137)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=ms, timezone=Indian/Kerguelen
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=2,free=162298040,total=490733568
[junit4] 2> NOTE: All tests run in this JVM: [SolrIndexConfigTest, TestAtomicUpdateErrorCases, OpenCloseCoreStressTest, TestSolr4Spatial, TestOmitPositions, HdfsCollectionsAPIDistributedZkTest, SimplePostToolTest, TestRecovery, TestQueryTypes, TestRandomMergePolicy, TestSolrXml, RequiredFieldsTest, DistanceFunctionTest, CollectionsAPIDistributedZkTest, XsltUpdateRequestHandlerTest, TestLazyCores, DirectSolrSpellCheckerTest, TestUniqueKeyFieldResource, StatsComponentTest, TestIndexSearcher, MinimalSchemaTest, TestComponentsName, StandardRequestHandlerTest, TestCursorMarkWithoutUniqueKey, TestUtils, TestArbitraryIndexDir, TestStressLucene, DirectUpdateHandlerTest, TestRecoveryHdfs, FullSolrCloudDistribCmdsTest, TestStressUserVersions, TimeZoneUtilsTest, NotRequiredUniqueKeyTest, HdfsLockFactoryTest, IndexBasedSpellCheckerTest, PrimUtilsTest, TestFieldResource, TestBinaryField, TestFileDictionaryLookup, BlockDirectoryTest, NoCacheHeaderTest, TestPartialUpdateDeduplication, ShardRoutingCustomTest, RequestHandlersTest, TestJmxIntegration, TestAnalyzedSuggestions, DocValuesMultiTest, TestSolrCoreProperties, TestCollationField, SortByFunctionTest, DocValuesMissingTest, TestFastLRUCache, TestSolrJ, CurrencyFieldXmlFileTest, TestStressRecovery, TestSchemaNameResource, CurrencyFieldOpenExchangeTest, BadIndexSchemaTest, TestAddFieldRealTimeGet, TestReload, TestWordDelimiterFilterFactory, TestDistribDocBasedVersion, OutputWriterTest, TestDistributedSearch, UniqFieldsUpdateProcessorFactoryTest, OpenExchangeRatesOrgProviderTest, SpellCheckComponentTest, TestFieldTypeResource, TestUpdate, TestDistributedGrouping, SolrCoreCheckLockOnStartupTest, HdfsSyncSliceTest, TestDFRSimilarityFactory, PreAnalyzedFieldTest, ChangedSchemaMergeTest, DateFieldTest, SignatureUpdateProcessorFactoryTest, TestBinaryResponseWriter, PluginInfoTest, TestMultiCoreConfBootstrap, TestCollationFieldDocValues, ZkSolrClientTest, SolrCmdDistributorTest, HighlighterConfigTest, MultiTermTest, TestCSVResponseWriter, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, BasicZkTest, RecoveryZkTest, TestRandomFaceting, TestZkChroot, TestRandomDVFaceting, ZkCLITest, SliceStateUpdateTest, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, ZkControllerTest, TestStressReorder, TestStressVersions, HardAutoCommitTest, TestRangeQuery, TestCoreContainer, SimpleFacetsTest, PeerSyncTest, ConvertedLegacyTest, TestFiltering, BasicFunctionalityTest, TestBadConfig, OverseerCollectionProcessorTest, HighlighterTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, SuggesterFSTTest, CoreAdminHandlerTest, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTSTTest, SuggesterTest, SpatialFilterTest, SuggesterWFSTTest, TestCSVLoader, WordBreakSolrSpellCheckerTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestRemoteStreaming, TestSolrDeletionPolicy1, DebugComponentTest, CacheHeaderTest, TestSurroundQueryParser, LukeRequestHandlerTest, DisMaxRequestHandlerTest, TestQueryUtils, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, JSONWriterTest, MBeansHandlerTest, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPHPSerializedResponseWriter, PingRequestHandlerTest, CSVRequestHandlerTest, SearchHandlerTest, CoreContainerCoreInitFailuresTest, AlternateDirectoryTest, TestSolrIndexConfig, BadComponentTest, TestMergePolicyConfig, TestElisionMultitermQuery, TestFuzzyAnalyzedSuggestions, TestPostingsSolrHighlighter, TestPhraseSuggestions, TestCodecSupport, EchoParamsTest, TestPerFieldSimilarity, TestIBSimilarityFactory, TestFastWriter, LegacyHTMLStripCharFilterTest, ChaosMonkeyNothingIsSafeTest]
[junit4] Completed on J1 in 309.93s, 1 test, 1 failure <<< FAILURES!
[...truncated 143 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:459: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:439: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/extra-targets.xml:37: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:491: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1308: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:941: There were test failures: 358 suites, 1552 tests, 1 failure, 35 ignored (7 assumptions)
Total time: 65 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure