You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2013/11/12 22:14:24 UTC
[JENKINS] Lucene-Solr-Tests-4.x-Java6 - Build # 2159 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-4.x-Java6/2159/
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard3 is not consistent. Got 48 from http://127.0.0.1:35294/m/collection1lastClient and got 41 from http://127.0.0.1:48408/m/collection1
Stack Trace:
java.lang.AssertionError: shard3 is not consistent. Got 48 from http://127.0.0.1:35294/m/collection1lastClient and got 41 from http://127.0.0.1:48408/m/collection1
at __randomizedtesting.SeedInfo.seed([B3FA599CBD89A4E6:321CD784CAD6C4DA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1151)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:135)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:679)
Build Log:
[...truncated 10008 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> 978069 T1621 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /m/
[junit4] 2> 978075 T1621 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1384289717312
[junit4] 2> 978077 T1621 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 978077 T1622 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 978178 T1621 oasc.ZkTestServer.run start zk server on port:43933
[junit4] 2> 978179 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 978208 T1628 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@659d7012 name:ZooKeeperConnection Watcher:127.0.0.1:43933 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 978209 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 978209 T1621 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 978288 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 978315 T1630 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22f4d1c4 name:ZooKeeperConnection Watcher:127.0.0.1:43933/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 978316 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 978316 T1621 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 978405 T1621 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 978503 T1621 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 978565 T1621 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 978583 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 978583 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 978680 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 978680 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 978792 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 978793 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 978801 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 978801 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 978808 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 978809 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 978818 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 978818 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 978832 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 978833 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 978843 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 978844 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 978853 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 978853 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 978864 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 978865 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 978872 T1621 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 978873 T1621 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 979072 T1621 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 979075 T1621 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:41315
[junit4] 2> 979076 T1621 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 979076 T1621 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 979077 T1621 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120
[junit4] 2> 979077 T1621 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120/'
[junit4] 2> 979115 T1621 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120/solr.xml
[junit4] 2> 979188 T1621 oasc.CoreContainer.<init> New CoreContainer 507393937
[junit4] 2> 979189 T1621 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120/]
[junit4] 2> 979191 T1621 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 979191 T1621 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 979191 T1621 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 979192 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 979192 T1621 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 979193 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 979193 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 979194 T1621 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 979194 T1621 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 979201 T1621 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 979201 T1621 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 979202 T1621 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43933/solr
[junit4] 2> 979202 T1621 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 979203 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 979212 T1641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b2201a name:ZooKeeperConnection Watcher:127.0.0.1:43933 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 979213 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 979228 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 979231 T1643 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b9fde52 name:ZooKeeperConnection Watcher:127.0.0.1:43933/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 979231 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 979234 T1621 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 979244 T1621 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 979248 T1621 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 979255 T1621 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41315_m
[junit4] 2> 979257 T1621 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41315_m
[junit4] 2> 979260 T1621 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 979279 T1621 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 979282 T1621 oasc.Overseer.start Overseer (id=90720810914152451-127.0.0.1:41315_m-n_0000000000) starting
[junit4] 2> 979293 T1621 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 979304 T1645 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 979305 T1621 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 979315 T1621 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 979317 T1621 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 979320 T1644 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 979327 T1646 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 979327 T1646 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 979329 T1646 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 980826 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 980826 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:41315/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41315_m",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 980827 T1644 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 980827 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 980839 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 981332 T1646 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 981332 T1646 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120/collection1
[junit4] 2> 981332 T1646 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 981333 T1646 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 981334 T1646 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 981335 T1646 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120/collection1/'
[junit4] 2> 981337 T1646 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120/collection1/lib/classes/' to classloader
[junit4] 2> 981337 T1646 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120/collection1/lib/README' to classloader
[junit4] 2> 981399 T1646 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 981461 T1646 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 981563 T1646 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 981574 T1646 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 982198 T1646 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 982199 T1646 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 982200 T1646 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 982212 T1646 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 982217 T1646 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 982232 T1646 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 982237 T1646 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 982242 T1646 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 982243 T1646 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 982243 T1646 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 982243 T1646 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 982245 T1646 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 982245 T1646 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 982245 T1646 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 982301 T1646 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1384289718120/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/control/data/
[junit4] 2> 982301 T1646 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ff61490
[junit4] 2> 982303 T1646 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/control/data
[junit4] 2> 982304 T1646 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/control/data/index/
[junit4] 2> 982305 T1646 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 982306 T1646 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/control/data/index
[junit4] 2> 982306 T1646 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 982311 T1646 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 982312 T1646 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 982315 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 982315 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 982316 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 982316 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 982316 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 982317 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 982317 T1646 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 982317 T1646 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 982318 T1646 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 982318 T1646 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 982319 T1646 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 982319 T1646 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 982320 T1646 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 982320 T1646 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 982321 T1646 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 982330 T1646 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 982334 T1646 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 982335 T1646 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 982336 T1646 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6868202160029517]
[junit4] 2> 982337 T1646 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 982338 T1646 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 982338 T1646 oass.SolrIndexSearcher.<init> Opening Searcher@3493d94a main
[junit4] 2> 982341 T1647 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3493d94a main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 982343 T1646 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 982343 T1646 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:41315/m collection:control_collection shard:shard1
[junit4] 2> 982345 T1646 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 982362 T1646 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 982373 T1646 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 982373 T1646 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 982374 T1646 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:41315/m/collection1/
[junit4] 2> 982374 T1646 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 982374 T1646 oasc.SyncStrategy.syncToMe http://127.0.0.1:41315/m/collection1/ has no replicas
[junit4] 2> 982374 T1646 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:41315/m/collection1/ shard1
[junit4] 2> 982375 T1646 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 983847 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 983868 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 983884 T1646 oasc.ZkController.register We are http://127.0.0.1:41315/m/collection1/ and leader is http://127.0.0.1:41315/m/collection1/
[junit4] 2> 983884 T1646 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41315/m
[junit4] 2> 983884 T1646 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 983884 T1646 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 983885 T1646 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 983886 T1646 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 983888 T1621 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 983888 T1621 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 983891 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 983893 T1650 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@288bae74 name:ZooKeeperConnection Watcher:127.0.0.1:43933/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 983893 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 983895 T1621 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 983897 T1621 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 984175 T1621 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 984178 T1621 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:27199
[junit4] 2> 984178 T1621 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 984179 T1621 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 984179 T1621 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134
[junit4] 2> 984180 T1621 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134/'
[junit4] 2> 984217 T1621 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134/solr.xml
[junit4] 2> 984289 T1621 oasc.CoreContainer.<init> New CoreContainer 1080062316
[junit4] 2> 984290 T1621 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134/]
[junit4] 2> 984292 T1621 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 984292 T1621 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 984293 T1621 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 984293 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 984294 T1621 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 984294 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 984295 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 984295 T1621 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 984296 T1621 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 984302 T1621 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 984303 T1621 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 984303 T1621 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43933/solr
[junit4] 2> 984304 T1621 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 984305 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 984308 T1661 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7bb68a78 name:ZooKeeperConnection Watcher:127.0.0.1:43933 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 984308 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 984319 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 984321 T1663 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61f548d9 name:ZooKeeperConnection Watcher:127.0.0.1:43933/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 984321 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 984334 T1621 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 985338 T1621 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:27199_m
[junit4] 2> 985341 T1621 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:27199_m
[junit4] 2> 985345 T1650 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 985345 T1663 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 985345 T1643 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 985364 T1664 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 985365 T1664 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 985366 T1664 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 985375 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 985376 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:41315/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41315_m",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 985380 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:27199/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:27199_m",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 985380 T1644 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 985381 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 985385 T1650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 985385 T1663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 985385 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 986367 T1664 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 986367 T1664 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134/collection1
[junit4] 2> 986367 T1664 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 986368 T1664 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 986368 T1664 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 986370 T1664 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134/collection1/'
[junit4] 2> 986371 T1664 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134/collection1/lib/classes/' to classloader
[junit4] 2> 986371 T1664 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134/collection1/lib/README' to classloader
[junit4] 2> 986432 T1664 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 986493 T1664 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 986596 T1664 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 986607 T1664 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 987228 T1664 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 987229 T1664 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 987230 T1664 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 987242 T1664 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 987247 T1664 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 987267 T1664 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 987272 T1664 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 987276 T1664 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 987278 T1664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 987278 T1664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 987278 T1664 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 987279 T1664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 987280 T1664 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 987280 T1664 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 987280 T1664 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1384289723134/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty1/
[junit4] 2> 987281 T1664 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ff61490
[junit4] 2> 987282 T1664 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty1
[junit4] 2> 987283 T1664 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty1/index/
[junit4] 2> 987284 T1664 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 987284 T1664 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty1/index
[junit4] 2> 987285 T1664 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 987289 T1664 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 987289 T1664 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 987293 T1664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 987294 T1664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 987294 T1664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 987294 T1664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 987295 T1664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 987295 T1664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 987296 T1664 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 987296 T1664 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 987296 T1664 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 987297 T1664 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 987297 T1664 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 987298 T1664 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 987298 T1664 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 987299 T1664 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 987299 T1664 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 987309 T1664 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 987313 T1664 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 987313 T1664 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 987314 T1664 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6868202160029517]
[junit4] 2> 987315 T1664 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 987316 T1664 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 987316 T1664 oass.SolrIndexSearcher.<init> Opening Searcher@79c1510d main
[junit4] 2> 987320 T1665 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79c1510d main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 987322 T1664 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 987323 T1664 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:27199/m collection:collection1 shard:shard1
[junit4] 2> 987324 T1664 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 987347 T1664 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 987355 T1664 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 987356 T1664 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 987356 T1664 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:27199/m/collection1/
[junit4] 2> 987356 T1664 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 987356 T1664 oasc.SyncStrategy.syncToMe http://127.0.0.1:27199/m/collection1/ has no replicas
[junit4] 2> 987357 T1664 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:27199/m/collection1/ shard1
[junit4] 2> 987357 T1664 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 988395 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 988422 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 988422 T1663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 988422 T1650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 988451 T1664 oasc.ZkController.register We are http://127.0.0.1:27199/m/collection1/ and leader is http://127.0.0.1:27199/m/collection1/
[junit4] 2> 988451 T1664 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:27199/m
[junit4] 2> 988451 T1664 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 988451 T1664 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 988452 T1664 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 988454 T1664 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 988455 T1621 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 988456 T1621 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 988737 T1621 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 988740 T1621 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:31396
[junit4] 2> 988740 T1621 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 988741 T1621 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 988741 T1621 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693
[junit4] 2> 988742 T1621 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693/'
[junit4] 2> 988779 T1621 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693/solr.xml
[junit4] 2> 988851 T1621 oasc.CoreContainer.<init> New CoreContainer 1262469625
[junit4] 2> 988852 T1621 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693/]
[junit4] 2> 988854 T1621 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 988854 T1621 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 988855 T1621 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 988855 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 988856 T1621 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 988856 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 988857 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 988857 T1621 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 988858 T1621 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 988864 T1621 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 988865 T1621 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 988865 T1621 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43933/solr
[junit4] 2> 988866 T1621 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 988867 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 988870 T1677 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46d7e907 name:ZooKeeperConnection Watcher:127.0.0.1:43933 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 988870 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 988890 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 988892 T1679 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b53aced name:ZooKeeperConnection Watcher:127.0.0.1:43933/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 988892 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 988897 T1621 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 989902 T1621 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31396_m
[junit4] 2> 989904 T1621 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31396_m
[junit4] 2> 989908 T1650 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 989908 T1643 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 989908 T1663 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 989908 T1679 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 989928 T1680 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 989928 T1680 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 989929 T1680 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 989930 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 989931 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:27199/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:27199_m",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 989941 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:31396/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:31396_m",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 989942 T1644 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 989942 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 989953 T1679 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 989953 T1663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 989953 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 989953 T1650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 990931 T1680 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 990931 T1680 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693/collection1
[junit4] 2> 990931 T1680 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 990932 T1680 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 990932 T1680 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 990934 T1680 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693/collection1/'
[junit4] 2> 990935 T1680 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693/collection1/lib/README' to classloader
[junit4] 2> 990936 T1680 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693/collection1/lib/classes/' to classloader
[junit4] 2> 990998 T1680 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 991059 T1680 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 991162 T1680 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 991173 T1680 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 991796 T1680 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 991796 T1680 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 991797 T1680 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 991810 T1680 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 991814 T1680 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 991829 T1680 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 991834 T1680 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 991838 T1680 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 991840 T1680 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 991840 T1680 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 991840 T1680 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 991841 T1680 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 991842 T1680 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 991842 T1680 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 991842 T1680 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../../usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1384289727693/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty2/
[junit4] 2> 991842 T1680 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ff61490
[junit4] 2> 991844 T1680 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty2
[junit4] 2> 991845 T1680 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty2/index/
[junit4] 2> 991845 T1680 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 991846 T1680 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty2/index
[junit4] 2> 991847 T1680 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 991851 T1680 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 991851 T1680 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 991855 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 991856 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 991856 T1680 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 991857 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 991857 T1680 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 991857 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 991858 T1680 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 991858 T1680 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 991858 T1680 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 991859 T1680 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 991860 T1680 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 991860 T1680 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 991860 T1680 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 991861 T1680 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 991861 T1680 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 991871 T1680 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 991875 T1680 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 991875 T1680 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 991876 T1680 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6868202160029517]
[junit4] 2> 991877 T1680 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 991878 T1680 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 991878 T1680 oass.SolrIndexSearcher.<init> Opening Searcher@3abcab10 main
[junit4] 2> 991883 T1681 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3abcab10 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 991885 T1680 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 991885 T1680 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:31396/m collection:collection1 shard:shard2
[junit4] 2> 991887 T1680 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 991905 T1680 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 991913 T1680 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 991914 T1680 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 991914 T1680 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:31396/m/collection1/
[junit4] 2> 991914 T1680 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 991915 T1680 oasc.SyncStrategy.syncToMe http://127.0.0.1:31396/m/collection1/ has no replicas
[junit4] 2> 991915 T1680 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:31396/m/collection1/ shard2
[junit4] 2> 991915 T1680 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 992963 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 992982 T1679 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 992982 T1663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 992982 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 992982 T1650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 993010 T1680 oasc.ZkController.register We are http://127.0.0.1:31396/m/collection1/ and leader is http://127.0.0.1:31396/m/collection1/
[junit4] 2> 993010 T1680 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:31396/m
[junit4] 2> 993010 T1680 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 993010 T1680 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 993011 T1680 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 993013 T1680 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 993014 T1621 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1
[junit4] 2> 993014 T1621 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 993300 T1621 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 993303 T1621 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:19907
[junit4] 2> 993303 T1621 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 993304 T1621 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 993304 T1621 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252
[junit4] 2> 993305 T1621 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252/'
[junit4] 2> 993342 T1621 oasc.ConfigSolr.fromFile Loading container configuration from /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252/solr.xml
[junit4] 2> 993415 T1621 oasc.CoreContainer.<init> New CoreContainer 1172796769
[junit4] 2> 993416 T1621 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252/]
[junit4] 2> 993417 T1621 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 993418 T1621 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 993418 T1621 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 993419 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 993419 T1621 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 993420 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 993420 T1621 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 993421 T1621 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 993421 T1621 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 993428 T1621 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 993428 T1621 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 993429 T1621 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43933/solr
[junit4] 2> 993429 T1621 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 993430 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 993450 T1693 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ecb5721 name:ZooKeeperConnection Watcher:127.0.0.1:43933 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 993451 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 993470 T1621 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 993472 T1695 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76a7e6e7 name:ZooKeeperConnection Watcher:127.0.0.1:43933/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 993472 T1621 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 993516 T1621 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 994496 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 994497 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:31396/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:31396_m",
[junit4] 2> "shard":"shard2",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 994516 T1679 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 994516 T1695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 994516 T1663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 994516 T1650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 994516 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 994520 T1621 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:19907_m
[junit4] 2> 994522 T1621 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:19907_m
[junit4] 2> 994530 T1679 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 994531 T1650 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 994531 T1663 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 994531 T1695 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 994531 T1643 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 994539 T1696 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 994540 T1696 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 994541 T1696 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 996024 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 996025 T1644 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:19907/m",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:19907_m",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 996025 T1644 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 996025 T1644 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 996041 T1679 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 996041 T1663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 996041 T1695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 996041 T1650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 996041 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 996544 T1696 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 996544 T1696 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252/collection1
[junit4] 2> 996544 T1696 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 996545 T1696 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 996545 T1696 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 996546 T1696 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252/collection1/'
[junit4] 2> 996548 T1696 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252/collection1/lib/README' to classloader
[junit4] 2> 996548 T1696 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252/collection1/lib/classes/' to classloader
[junit4] 2> 996609 T1696 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_47
[junit4] 2> 996669 T1696 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 996772 T1696 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 996783 T1696 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 997412 T1696 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 997413 T1696 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
[junit4] 2> 997414 T1696 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 997426 T1696 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 997431 T1696 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 997446 T1696 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 997450 T1696 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 997455 T1696 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 997456 T1696 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 997456 T1696 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 997457 T1696 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 997458 T1696 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 997458 T1696 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 997458 T1696 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 997459 T1696 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1384289732252/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty3/
[junit4] 2> 997459 T1696 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6ff61490
[junit4] 2> 997460 T1696 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty3
[junit4] 2> 997461 T1696 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty3/index/
[junit4] 2> 997462 T1696 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 997463 T1696 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty3/index
[junit4] 2> 997464 T1696 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=38, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 997468 T1696 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty3/index,segFN=segments_1,generation=1}
[junit4] 2> 997468 T1696 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 997473 T1696 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 997473 T1696 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 997473 T1696 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 997474 T1696 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 997474 T1696 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 997474 T1696 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 997475 T1696 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 997475 T1696 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 997475 T1696 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 997476 T1696 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 997477 T1696 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 997477 T1696 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 997477 T1696 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 997478 T1696 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 997479 T1696 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 997488 T1696 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 997492 T1696 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 997492 T1696 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 997493 T1696 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=19, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6868202160029517]
[junit4] 2> 997495 T1696 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty3/index,segFN=segments_1,generation=1}
[junit4] 2> 997495 T1696 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 997496 T1696 oass.SolrIndexSearcher.<init> Opening Searcher@3a94040 main
[junit4] 2> 997500 T1697 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a94040 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 997502 T1696 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 997503 T1696 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:19907/m collection:collection1 shard:shard3
[junit4] 2> 997504 T1696 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 997521 T1696 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 997532 T1696 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 997532 T1696 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 997533 T1696 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:19907/m/collection1/
[junit4] 2> 997533 T1696 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 997533 T1696 oasc.SyncStrategy.syncToMe http://127.0.0.1:19907/m/collection1/ has no replicas
[junit4] 2> 997533 T1696 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:19907/m/collection1/ shard3
[junit4] 2> 997534 T1696 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 997548 T1644 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 997564 T1679 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 997564 T1663 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 997564 T1643 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 997564 T1695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 997564 T1650 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 997599 T1696 oasc.ZkController.register We are http://127.0.0.1:19907/m/collection1/ and leader is http://127.0.0.1:19907/m/collection1/
[junit4] 2> 997599 T1696 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:19907/m
[junit4]
[...truncated too long message...]
.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:366)
[junit4] 2> ... 11 more
[junit4] 2> Caused by: java.net.SocketTimeoutException: connect timed out
[junit4] 2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4] 2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
[junit4] 2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
[junit4] 2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
[junit4] 2> at java.net.Socket.connect(Socket.java:546)
[junit4] 2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4] 2> ... 19 more
[junit4] 2>
[junit4] 2> 1141851 T1854 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:48408/m DONE. sync failed
[junit4] 2> 1141851 T1854 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4] 2> 1141852 T1854 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 1141852 T1854 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@1e9f6308
[junit4] 2> 1141854 T1854 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=32,autocommits=0,soft autocommit maxTime=1000ms,soft autocommits=10,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=91,cumulative_deletesById=43,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4] 2> 1141854 T1854 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 1141855 T1854 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 1141855 T1854 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 1141856 T1854 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 1141857 T1854 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 1141858 T1854 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty12/index [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty12/index;done=false>>]
[junit4] 2> 1141858 T1854 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty12/index
[junit4] 2> 1141858 T1854 oasc.CachingDirectoryFactory.closeCacheValue looking to close /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty12 [CachedDir<<refCount=0;path=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty12;done=false>>]
[junit4] 2> 1141858 T1854 oasc.CachingDirectoryFactory.close Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1384289717312/jetty12
[junit4] 2> 1141859 T1854 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1141859 T1854 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1141859 T1854 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1141859 T1854 oasc.LeaderElector$1.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:247)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:244)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:244)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4] 2>
[junit4] 2> NOTE: test params are: codec=Lucene46: {range_facet_l=PostingsFormat(name=TestBloomFilteredLucene41Postings), text=MockFixedIntBlock(blockSize=824), _version_=PostingsFormat(name=TestBloomFilteredLucene41Postings), rnd_b=MockVariableIntBlock(baseBlockSize=57), intDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), id=MockVariableIntBlock(baseBlockSize=57), timestamp=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_t=PostingsFormat(name=SimpleText), range_facet_sl=MockVariableIntBlock(baseBlockSize=57), range_facet_si=PostingsFormat(name=SimpleText), other_tl1=PostingsFormat(name=TestBloomFilteredLucene41Postings), multiDefault=PostingsFormat(name=TestBloomFilteredLucene41Postings), a_si=PostingsFormat(name=TestBloomFilteredLucene41Postings)}, docValues:{}, sim=DefaultSimilarity, locale=ja_JP, timezone=Africa/Tunis
[junit4] 2> NOTE: FreeBSD 9.1-RELEASE-p3 amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=3,free=124150664,total=419299328
[junit4] 2> NOTE: All tests run in this JVM: [TestCoreDiscovery, HighlighterTest, StatsComponentTest, SuggesterWFSTTest, UniqFieldsUpdateProcessorFactoryTest, TestIBSimilarityFactory, TestZkChroot, DateFieldTest, QueryResultKeyTest, DistanceFunctionTest, DisMaxRequestHandlerTest, TestFieldResource, PathHierarchyTokenizerFactoryTest, HdfsLockFactoryTest, OutputWriterTest, TestSerializedLuceneMatchVersion, TestUpdate, TestIndexingPerformance, TestQuerySenderListener, TestWordDelimiterFilterFactory, HighlighterConfigTest, ConvertedLegacyTest, MultiTermTest, SignatureUpdateProcessorFactoryTest, TestXIncludeConfig, DebugComponentTest, TestStressUserVersions, TestMultiCoreConfBootstrap, CoreMergeIndexesAdminHandlerTest, SuggesterTSTTest, TestShardHandlerFactory, SolrCoreCheckLockOnStartupTest, TestReload, AddSchemaFieldsUpdateProcessorFactoryTest, ZkNodePropsTest, RAMDirectoryFactoryTest, OpenExchangeRatesOrgProviderTest, RequiredFieldsTest, TestManagedSchemaFieldResource, MinimalSchemaTest, LeaderElectionIntegrationTest, HdfsBasicDistributedZkTest, TestOmitPositions, TestCollationKeyRangeQueries, TestRandomFaceting, SolrCmdDistributorTest, TestDynamicFieldResource, OpenCloseCoreStressTest, StandardRequestHandlerTest, TestJmxIntegration, HardAutoCommitTest, TestFastOutputStream, TestDistributedGrouping, TestSchemaResource, DefaultValueUpdateProcessorTest, TestCollationField, TestCSVLoader, ZkSolrClientTest, PingRequestHandlerTest, TestUtils, BasicZkTest, DeleteInactiveReplicaTest, FullSolrCloudDistribCmdsTest, TestComponentsName, TestFastWriter, SolrPluginUtilsTest, BadIndexSchemaTest, TestSolrCoreProperties, SpellingQueryConverterTest, TestArbitraryIndexDir, PreAnalyzedUpdateProcessorTest, TestBinaryResponseWriter, TestAnalyzedSuggestions, TestRangeQuery, DirectSolrSpellCheckerTest, TestCSVResponseWriter, FileUtilsTest, TestCloudManagedSchemaAddField, FieldAnalysisRequestHandlerTest, TestStressRecovery, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 164.75s, 1 test, 1 failure <<< FAILURES!
[...truncated 528 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:428: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:408: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/build.xml:189: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/solr/common-build.xml:501: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:1276: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-Tests-4.x-Java6/lucene/common-build.xml:912: There were test failures: 335 suites, 1475 tests, 1 failure, 35 ignored (6 assumptions)
Total time: 63 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure