You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/11/03 20:27:37 UTC
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_45) - Build #
8160 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8160/
Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.testDistribSearch
Error Message:
document count mismatch. control=52 sum(shards)=53 cloudClient=53
Stack Trace:
java.lang.AssertionError: document count mismatch. control=52 sum(shards)=53 cloudClient=53
at __randomizedtesting.SeedInfo.seed([1619C907BDBD6EC1:97FF471FCAE20EFD]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1240)
at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:194)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
at java.lang.Thread.run(Thread.java:744)
Build Log:
[...truncated 10573 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest
[junit4] 2> 1143633 T3513 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 1143637 T3513 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeyNothingIsSafeTest-1383506636837
[junit4] 2> 1143638 T3513 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 1143638 T3514 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 1143738 T3513 oasc.ZkTestServer.run start zk server on port:34620
[junit4] 2> 1143739 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1143838 T3520 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56c84f75 name:ZooKeeperConnection Watcher:127.0.0.1:34620 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1143838 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1143838 T3513 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 1143845 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1143848 T3522 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@22bfa9e6 name:ZooKeeperConnection Watcher:127.0.0.1:34620/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1143848 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1143849 T3513 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 1143853 T3513 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 1143857 T3513 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 1143860 T3513 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 1143864 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1143865 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 1143870 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 1143871 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 1143876 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1143877 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1143881 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1143881 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 1143886 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1143886 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 1143890 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1143891 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 1143895 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1143896 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 1143900 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1143900 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1143905 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1143906 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 1143911 T3513 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1143912 T3513 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 1143995 T3513 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1144010 T3513 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42041
[junit4] 2> 1144011 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1144011 T3513 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1144012 T3513 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116
[junit4] 2> 1144012 T3513 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116/'
[junit4] 2> 1144026 T3513 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116/solr.xml
[junit4] 2> 1144065 T3513 oasc.CoreContainer.<init> New CoreContainer 803075766
[junit4] 2> 1144066 T3513 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116/]
[junit4] 2> 1144067 T3513 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1144067 T3513 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1144067 T3513 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1144067 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1144068 T3513 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1144068 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1144068 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1144068 T3513 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1144068 T3513 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1144069 T3513 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1144070 T3513 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1144070 T3513 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34620/solr
[junit4] 2> 1144070 T3513 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1144071 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1144073 T3533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@752d9148 name:ZooKeeperConnection Watcher:127.0.0.1:34620 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1144074 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1144077 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1144078 T3535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ca0560b name:ZooKeeperConnection Watcher:127.0.0.1:34620/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1144078 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1144080 T3513 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 1144082 T3513 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 1144085 T3513 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 1144086 T3513 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42041_
[junit4] 2> 1144087 T3513 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42041_
[junit4] 2> 1144089 T3513 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 1144093 T3513 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 1144095 T3513 oasc.Overseer.start Overseer (id=90669490951880707-127.0.0.1:42041_-n_0000000000) starting
[junit4] 2> 1144100 T3513 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 1144103 T3537 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 1144104 T3513 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 1144106 T3513 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 1144108 T3513 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1144110 T3536 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 1144112 T3538 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1144112 T3538 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1144113 T3538 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1145613 T3536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1145614 T3536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:42041",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42041_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1145614 T3536 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 1145614 T3536 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1145620 T3535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1146114 T3538 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1146114 T3538 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116/collection1
[junit4] 2> 1146114 T3538 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 1146115 T3538 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1146116 T3538 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 1146117 T3538 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116/collection1/'
[junit4] 2> 1146118 T3538 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116/collection1/lib/classes/' to classloader
[junit4] 2> 1146118 T3538 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116/collection1/lib/README' to classloader
[junit4] 2> 1146157 T3538 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1146206 T3538 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1146207 T3538 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1146235 T3538 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1146598 T3538 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1146603 T3538 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1146605 T3538 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1146620 T3538 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1146625 T3538 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1146629 T3538 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1146631 T3538 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1146631 T3538 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1146631 T3538 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1146632 T3538 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1146633 T3538 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1146633 T3538 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1146633 T3538 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-controljetty-1383506637116/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/control/data/
[junit4] 2> 1146633 T3538 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5731ac04
[junit4] 2> 1146634 T3538 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/control/data
[junit4] 2> 1146634 T3538 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/control/data/index/
[junit4] 2> 1146635 T3538 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 1146635 T3538 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/control/data/index
[junit4] 2> 1146636 T3538 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=79.3544921875, floorSegmentMB=1.814453125, forceMergeDeletesPctAllowed=4.415637898671342, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.160050807901862
[junit4] 2> 1146638 T3538 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1146638 T3538 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1146641 T3538 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1146641 T3538 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1146641 T3538 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1146641 T3538 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1146641 T3538 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1146642 T3538 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1146642 T3538 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1146642 T3538 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1146642 T3538 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1146643 T3538 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1146643 T3538 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1146643 T3538 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1146643 T3538 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1146644 T3538 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1146644 T3538 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1146646 T3538 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1146648 T3538 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1146648 T3538 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1146649 T3538 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=45.4609375, floorSegmentMB=0.392578125, forceMergeDeletesPctAllowed=2.0437853862737274, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1146650 T3538 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 1146650 T3538 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1146650 T3538 oass.SolrIndexSearcher.<init> Opening Searcher@51ae8e46 main
[junit4] 2> 1146652 T3539 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@51ae8e46 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1146653 T3538 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1146653 T3538 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42041 collection:control_collection shard:shard1
[junit4] 2> 1146654 T3538 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 1146659 T3538 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1146660 T3538 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1146661 T3538 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1146661 T3538 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42041/collection1/
[junit4] 2> 1146661 T3538 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1146661 T3538 oasc.SyncStrategy.syncToMe http://127.0.0.1:42041/collection1/ has no replicas
[junit4] 2> 1146661 T3538 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42041/collection1/ shard1
[junit4] 2> 1146661 T3538 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 1147126 T3536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1147137 T3535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 1147167 T3538 oasc.ZkController.register We are http://127.0.0.1:42041/collection1/ and leader is http://127.0.0.1:42041/collection1/
[junit4] 2> 1147167 T3538 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42041
[junit4] 2> 1147168 T3538 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1147168 T3538 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1147168 T3538 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1147171 T3538 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1147173 T3513 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 1147173 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1147176 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1147178 T3542 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50075c8b name:ZooKeeperConnection Watcher:127.0.0.1:34620/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1147179 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1147181 T3513 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1147184 T3513 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1147259 T3513 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1147261 T3513 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37955
[junit4] 2> 1147262 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1147262 T3513 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1147263 T3513 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384
[junit4] 2> 1147263 T3513 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384/'
[junit4] 2> 1147276 T3513 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384/solr.xml
[junit4] 2> 1147308 T3513 oasc.CoreContainer.<init> New CoreContainer 1000418306
[junit4] 2> 1147309 T3513 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384/]
[junit4] 2> 1147310 T3513 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1147310 T3513 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1147310 T3513 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1147310 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1147310 T3513 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1147311 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1147311 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1147311 T3513 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1147311 T3513 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1147312 T3513 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1147313 T3513 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1147313 T3513 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34620/solr
[junit4] 2> 1147313 T3513 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1147314 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1147316 T3553 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37ac17a6 name:ZooKeeperConnection Watcher:127.0.0.1:34620 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1147316 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1147319 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1147320 T3555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73830947 name:ZooKeeperConnection Watcher:127.0.0.1:34620/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1147320 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1147324 T3513 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1148327 T3513 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37955_
[junit4] 2> 1148329 T3513 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37955_
[junit4] 2> 1148334 T3542 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1148334 T3535 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1148335 T3555 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 1148346 T3556 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1148346 T3556 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1148349 T3556 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1148645 T3536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1148645 T3536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:42041",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42041_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1148648 T3536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:37955",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37955_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1148648 T3536 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 1148648 T3536 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1148652 T3555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1148652 T3535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1148654 T3542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1149349 T3556 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1149349 T3556 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384/collection1
[junit4] 2> 1149349 T3556 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1149350 T3556 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1149350 T3556 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1149351 T3556 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384/collection1/'
[junit4] 2> 1149352 T3556 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384/collection1/lib/classes/' to classloader
[junit4] 2> 1149352 T3556 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384/collection1/lib/README' to classloader
[junit4] 2> 1149414 T3556 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1149447 T3556 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1149448 T3556 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1149479 T3556 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1149910 T3556 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1149915 T3556 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1149918 T3556 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1149929 T3556 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1149932 T3556 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1149936 T3556 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1149937 T3556 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1149938 T3556 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1149938 T3556 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1149939 T3556 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1149939 T3556 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1149939 T3556 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1149940 T3556 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty1-1383506640384/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty1/
[junit4] 2> 1149940 T3556 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5731ac04
[junit4] 2> 1149941 T3556 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty1
[junit4] 2> 1149941 T3556 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty1/index/
[junit4] 2> 1149941 T3556 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 1149941 T3556 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty1/index
[junit4] 2> 1149942 T3556 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=79.3544921875, floorSegmentMB=1.814453125, forceMergeDeletesPctAllowed=4.415637898671342, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.160050807901862
[junit4] 2> 1149944 T3556 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1149944 T3556 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1149948 T3556 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1149949 T3556 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1149949 T3556 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1149949 T3556 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1149950 T3556 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1149950 T3556 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1149950 T3556 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1149951 T3556 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1149951 T3556 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1149951 T3556 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1149952 T3556 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1149952 T3556 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1149952 T3556 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1149953 T3556 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1149953 T3556 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1149955 T3556 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1149957 T3556 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1149957 T3556 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1149958 T3556 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=45.4609375, floorSegmentMB=0.392578125, forceMergeDeletesPctAllowed=2.0437853862737274, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1149959 T3556 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 1149959 T3556 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1149959 T3556 oass.SolrIndexSearcher.<init> Opening Searcher@1a9b01d0 main
[junit4] 2> 1149962 T3557 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1a9b01d0 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1149964 T3556 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1149965 T3556 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:37955 collection:collection1 shard:shard1
[junit4] 2> 1149966 T3556 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 1149971 T3556 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 1149972 T3556 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1149973 T3556 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1149973 T3556 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:37955/collection1/
[junit4] 2> 1149973 T3556 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1149973 T3556 oasc.SyncStrategy.syncToMe http://127.0.0.1:37955/collection1/ has no replicas
[junit4] 2> 1149973 T3556 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:37955/collection1/ shard1
[junit4] 2> 1149973 T3556 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 1150155 T3536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1150161 T3555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1150162 T3542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1150161 T3535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 1150182 T3556 oasc.ZkController.register We are http://127.0.0.1:37955/collection1/ and leader is http://127.0.0.1:37955/collection1/
[junit4] 2> 1150182 T3556 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:37955
[junit4] 2> 1150182 T3556 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1150182 T3556 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1150183 T3556 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1150184 T3556 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1150185 T3513 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 1150186 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1150285 T3513 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1150287 T3513 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33666
[junit4] 2> 1150288 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1150289 T3513 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1150289 T3513 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386
[junit4] 2> 1150289 T3513 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386/'
[junit4] 2> 1150301 T3513 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386/solr.xml
[junit4] 2> 1150366 T3513 oasc.CoreContainer.<init> New CoreContainer 1413595540
[junit4] 2> 1150366 T3513 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386/]
[junit4] 2> 1150367 T3513 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1150368 T3513 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1150368 T3513 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1150368 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1150368 T3513 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1150369 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1150369 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1150369 T3513 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1150370 T3513 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1150371 T3513 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1150371 T3513 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1150372 T3513 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34620/solr
[junit4] 2> 1150372 T3513 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1150373 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1150375 T3569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3a42bf6d name:ZooKeeperConnection Watcher:127.0.0.1:34620 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1150375 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1150377 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1150379 T3571 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1459fd3b name:ZooKeeperConnection Watcher:127.0.0.1:34620/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1150379 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1150383 T3513 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1151386 T3513 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33666_
[junit4] 2> 1151387 T3513 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33666_
[junit4] 2> 1151390 T3555 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1151390 T3535 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1151390 T3542 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1151390 T3571 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 1151396 T3572 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1151396 T3572 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1151398 T3572 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1151666 T3536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1151667 T3536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:37955",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:37955_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 1151673 T3536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:33666",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:33666_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1151673 T3536 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1151674 T3536 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 1151679 T3555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1151679 T3535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1151679 T3571 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1151679 T3542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1152399 T3572 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1152399 T3572 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386/collection1
[junit4] 2> 1152399 T3572 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1152400 T3572 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1152400 T3572 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1152401 T3572 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386/collection1/'
[junit4] 2> 1152402 T3572 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386/collection1/lib/classes/' to classloader
[junit4] 2> 1152403 T3572 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386/collection1/lib/README' to classloader
[junit4] 2> 1152451 T3572 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1152498 T3572 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1152499 T3572 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1152527 T3572 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1152908 T3572 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1152915 T3572 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1152918 T3572 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1152929 T3572 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1152933 T3572 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1152936 T3572 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1152937 T3572 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1152938 T3572 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1152938 T3572 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1152939 T3572 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1152939 T3572 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1152939 T3572 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1152939 T3572 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty2-1383506643386/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty2/
[junit4] 2> 1152940 T3572 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5731ac04
[junit4] 2> 1152940 T3572 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty2
[junit4] 2> 1152941 T3572 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty2/index/
[junit4] 2> 1152941 T3572 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 1152941 T3572 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty2/index
[junit4] 2> 1152942 T3572 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=79.3544921875, floorSegmentMB=1.814453125, forceMergeDeletesPctAllowed=4.415637898671342, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.160050807901862
[junit4] 2> 1152944 T3572 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1152944 T3572 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1152947 T3572 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1152948 T3572 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1152948 T3572 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1152948 T3572 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1152949 T3572 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1152949 T3572 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1152949 T3572 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1152950 T3572 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1152950 T3572 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1152951 T3572 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1152951 T3572 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1152952 T3572 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1152952 T3572 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1152953 T3572 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1152953 T3572 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1152955 T3572 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1152958 T3572 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1152958 T3572 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1152959 T3572 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=45.4609375, floorSegmentMB=0.392578125, forceMergeDeletesPctAllowed=2.0437853862737274, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1152960 T3572 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 1152961 T3572 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1152961 T3572 oass.SolrIndexSearcher.<init> Opening Searcher@6d25b9d main
[junit4] 2> 1152965 T3573 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d25b9d main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1152966 T3572 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1152966 T3572 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33666 collection:collection1 shard:shard2
[junit4] 2> 1152967 T3572 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 1152972 T3572 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 1152974 T3572 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 1152975 T3572 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 1152975 T3572 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33666/collection1/
[junit4] 2> 1152975 T3572 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 1152975 T3572 oasc.SyncStrategy.syncToMe http://127.0.0.1:33666/collection1/ has no replicas
[junit4] 2> 1152975 T3572 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33666/collection1/ shard2
[junit4] 2> 1152975 T3572 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1153188 T3536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1153195 T3542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1153195 T3555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1153195 T3571 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1153195 T3535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 1153231 T3572 oasc.ZkController.register We are http://127.0.0.1:33666/collection1/ and leader is http://127.0.0.1:33666/collection1/
[junit4] 2> 1153231 T3572 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33666
[junit4] 2> 1153231 T3572 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 1153231 T3572 oasc.ZkController.publish publishing core=collection1 state=active
[junit4] 2> 1153231 T3572 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1153233 T3572 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1153234 T3513 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 1153234 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1153302 T3513 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1153304 T3513 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47300
[junit4] 2> 1153305 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1153305 T3513 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1153306 T3513 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434
[junit4] 2> 1153306 T3513 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434/'
[junit4] 2> 1153324 T3513 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434/solr.xml
[junit4] 2> 1153368 T3513 oasc.CoreContainer.<init> New CoreContainer 1144241126
[junit4] 2> 1153368 T3513 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434/]
[junit4] 2> 1153370 T3513 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1153370 T3513 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1153370 T3513 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1153371 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1153371 T3513 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1153371 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1153371 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1153371 T3513 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1153372 T3513 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1153373 T3513 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1153373 T3513 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1153373 T3513 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34620/solr
[junit4] 2> 1153374 T3513 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1153374 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1153376 T3585 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@570b78f2 name:ZooKeeperConnection Watcher:127.0.0.1:34620 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1153376 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1153378 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1153379 T3587 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@26bb5cdb name:ZooKeeperConnection Watcher:127.0.0.1:34620/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1153380 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1153388 T3513 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1154391 T3513 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47300_
[junit4] 2> 1154393 T3513 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47300_
[junit4] 2> 1154396 T3555 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1154397 T3542 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1154397 T3535 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1154397 T3587 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1154397 T3571 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4] 2> 1154404 T3588 oasc.ZkController.publish publishing core=collection1 state=down
[junit4] 2> 1154404 T3588 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1154405 T3588 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 1154702 T3536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1154703 T3536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:33666",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:33666_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 1154709 T3536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:47300",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47300_",
[junit4] 2> "shard":null,
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 1154709 T3536 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 1154710 T3536 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 1154716 T3555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1154717 T3542 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1154717 T3587 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1154716 T3571 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1154717 T3535 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 1155405 T3588 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 1155406 T3588 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434/collection1
[junit4] 2> 1155406 T3588 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 1155407 T3588 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 1155408 T3588 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 1155410 T3588 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434/collection1/'
[junit4] 2> 1155411 T3588 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434/collection1/lib/classes/' to classloader
[junit4] 2> 1155412 T3588 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434/collection1/lib/README' to classloader
[junit4] 2> 1155459 T3588 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4] 2> 1155515 T3588 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 1155516 T3588 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 1155529 T3588 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 1155918 T3588 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4] 2> 1155923 T3588 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 1155926 T3588 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 1155936 T3588 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1155940 T3588 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 1155943 T3588 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1155944 T3588 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1155944 T3588 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1155944 T3588 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4] 2> 1155945 T3588 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4] 2> 1155945 T3588 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4] 2> 1155946 T3588 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 1155946 T3588 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty3-1383506646434/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty3/
[junit4] 2> 1155946 T3588 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5731ac04
[junit4] 2> 1155947 T3588 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty3
[junit4] 2> 1155947 T3588 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty3/index/
[junit4] 2> 1155947 T3588 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty3/index' doesn't exist. Creating new index...
[junit4] 2> 1155948 T3588 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty3/index
[junit4] 2> 1155948 T3588 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=79.3544921875, floorSegmentMB=1.814453125, forceMergeDeletesPctAllowed=4.415637898671342, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.160050807901862
[junit4] 2> 1155950 T3588 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty3/index,segFN=segments_1,generation=1}
[junit4] 2> 1155950 T3588 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1155953 T3588 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 1155954 T3588 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 1155954 T3588 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 1155954 T3588 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 1155954 T3588 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 1155954 T3588 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 1155955 T3588 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1155955 T3588 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 1155955 T3588 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 1155956 T3588 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 1155956 T3588 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 1155956 T3588 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 1155956 T3588 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 1155957 T3588 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 1155957 T3588 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 1155960 T3588 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 1155962 T3588 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 1155963 T3588 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 1155963 T3588 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=17, maxMergedSegmentMB=45.4609375, floorSegmentMB=0.392578125, forceMergeDeletesPctAllowed=2.0437853862737274, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1155965 T3588 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-1383506636836/jetty3/index,segFN=segments_1,generation=1}
[junit4] 2> 1155965 T3588 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 1155965 T3588 oass.SolrIndexSearcher.<init> Opening Searcher@20da47d8 main
[junit4] 2> 1155968 T3589 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@20da47d8 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 1155970 T3588 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 1155971 T3588 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47300 collection:collection1 shard:shard1
[junit4] 2> 1155974 T3588 oasc.ZkController.register We are http://127.0.0.1:47300/collection1/ and leader is http://127.0.0.1:37955/collection1/
[junit4] 2> 1155974 T3588 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47300
[junit4] 2> 1155974 T3588 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4] 2> 1155974 T3588 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4] 2> ASYNC NEW_CORE C4159 name=collection1 org.apache.solr.core.SolrCore@3ec9556e url=http://127.0.0.1:47300/collection1 node=127.0.0.1:47300_ C4159_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:47300, core=collection1, node_name=127.0.0.1:47300_}
[junit4] 2> 1155975 T3590 C4159 P47300 oasc.RecoveryStrategy.run Starting recovery process. core=collection1 recoveringAfterStartup=true
[junit4] 2> 1155975 T3588 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1155975 T3590 C4159 P47300 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4] 2> 1155975 T3590 C4159 P47300 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4] 2> 1155975 T3590 C4159 P47300 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 1155976 T3513 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 1155976 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 1155978 T3550 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4] 2> 1156043 T3513 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 1156045 T3513 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35205
[junit4] 2> 1156045 T3513 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 1156046 T3513 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 1156047 T3513 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1383506649176
[junit4] 2> 1156047 T3513 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1383506649176/'
[junit4] 2> 1156061 T3513 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1383506649176/solr.xml
[junit4] 2> 1156111 T3513 oasc.CoreContainer.<init> New CoreContainer 11378174
[junit4] 2> 1156111 T3513 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest-jetty4-1383506649176/]
[junit4] 2> 1156113 T3513 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 1156113 T3513 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4] 2> 1156113 T3513 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 1156114 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 1156114 T3513 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 1156114 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 1156114 T3513 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 1156115 T3513 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 1156115 T3513 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 1156116 T3513 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 1156117 T3513 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 1156117 T3513 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:34620/solr
[junit4] 2> 1156118 T3513 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 1156119 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1156120 T3602 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6a66c3ee name:ZooKeeperConnection Watcher:127.0.0.1:34620 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1156120 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1156123 T3513 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 1156124 T3604 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25e487fc name:ZooKeeperConnection Watcher:127.0.0.1:34620/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 1156125 T3513 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 1156132 T3513 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 1156228 T3536 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 1156229 T3536 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "base_url":"http://127.0.0.1:47300",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:47300_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "shard_range":null,
[junit4] 2> "shard_state":"active",
[junit4] 2> "shard_parent":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2>
[...truncated too long message...]
[junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/90669490951880718-core_node5-n_0000000002 (0)
[junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/90669490951880722-core_node7-n_0000000003 (0)
[junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/90669490951880710-core_node1-n_0000000000 (0)
[junit4] 1> /solr/collections/collection1/leader_elect/shard1/election/90669490951880714-core_node3-n_0000000001 (0)
[junit4] 1> /solr/collections/collection1/leader_elect/shard2 (1)
[junit4] 1> /solr/collections/collection1/leader_elect/shard2/election (2)
[junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/90669490951880720-core_node6-n_0000000002 (0)
[junit4] 1> /solr/collections/collection1/leader_elect/shard2/election/90669490951880716-core_node4-n_0000000001 (0)
[junit4] 1> /solr/collections/collection1/leaders (2)
[junit4] 1> /solr/collections/collection1/leaders/shard1 (0)
[junit4] 1> DATA:
[junit4] 1> {
[junit4] 1> "core":"collection1",
[junit4] 1> "node_name":"127.0.0.1:37955_",
[junit4] 1> "base_url":"http://127.0.0.1:37955"}
[junit4] 1> /solr/collections/collection1/leaders/shard2 (0)
[junit4] 1> DATA:
[junit4] 1> {
[junit4] 1> "core":"collection1",
[junit4] 1> "node_name":"127.0.0.1:35205_",
[junit4] 1> "base_url":"http://127.0.0.1:35205"}
[junit4] 1> /solr/collections/control_collection (3)
[junit4] 1> DATA:
[junit4] 1> {"configName":"conf1"}
[junit4] 1> /solr/collections/control_collection/shards (0)
[junit4] 1> /solr/collections/control_collection/leader_elect (1)
[junit4] 1> /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4] 1> /solr/collections/control_collection/leader_elect/shard1/election/90669490951880707-core_node1-n_0000000000 (0)
[junit4] 1> /solr/collections/control_collection/leaders (1)
[junit4] 1> /solr/collections/control_collection/leaders/shard1 (0)
[junit4] 1> DATA:
[junit4] 1> {
[junit4] 1> "core":"collection1",
[junit4] 1> "node_name":"127.0.0.1:42041_",
[junit4] 1> "base_url":"http://127.0.0.1:42041"}
[junit4] 1> /solr/overseer_elect (2)
[junit4] 1> /solr/overseer_elect/election (7)
[junit4] 1> /solr/overseer_elect/election/90669490951880714-127.0.0.1:47300_-n_0000000003 (0)
[junit4] 1> /solr/overseer_elect/election/90669490951880707-127.0.0.1:42041_-n_0000000000 (0)
[junit4] 1> /solr/overseer_elect/election/90669490951880720-127.0.0.1:44218_-n_0000000006 (0)
[junit4] 1> /solr/overseer_elect/election/90669490951880722-127.0.0.1:58213_-n_0000000007 (0)
[junit4] 1> /solr/overseer_elect/election/90669490951880710-127.0.0.1:37955_-n_0000000001 (0)
[junit4] 1> /solr/overseer_elect/election/90669490951880716-127.0.0.1:35205_-n_0000000004 (0)
[junit4] 1> /solr/overseer_elect/election/90669490951880718-127.0.0.1:42270_-n_0000000005 (0)
[junit4] 1> /solr/overseer_elect/leader (0)
[junit4] 1> DATA:
[junit4] 1> {"id":"90669490951880707-127.0.0.1:42041_-n_0000000000"}
[junit4] 1> /zookeeper (1)
[junit4] 1> DATA:
[junit4] 1>
[junit4] 1>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyNothingIsSafeTest -Dtests.method=testDistribSearch -Dtests.seed=1619C907BDBD6EC1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PR -Dtests.timezone=Pacific/Saipan -Dtests.file.encoding=UTF-8
[junit4] FAILURE 51.3s J1 | ChaosMonkeyNothingIsSafeTest.testDistribSearch <<<
[junit4] > Throwable #1: java.lang.AssertionError: document count mismatch. control=52 sum(shards)=53 cloudClient=53
[junit4] > at __randomizedtesting.SeedInfo.seed([1619C907BDBD6EC1:97FF471FCAE20EFD]:0)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1240)
[junit4] > at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.doTest(ChaosMonkeyNothingIsSafeTest.java:194)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4] > at java.lang.Thread.run(Thread.java:744)
[junit4] 2> 1194981 T3513 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> 51353 T3512 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> 1195001 T3639 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1196099 T3658 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=es_PR, timezone=Pacific/Saipan
[junit4] 2> NOTE: Linux 3.8.0-32-generic amd64/Oracle Corporation 1.7.0_45 (64-bit)/cpus=8,threads=3,free=54841736,total=341209088
[junit4] 2> NOTE: All tests run in this JVM: [MultiTermTest, PrimUtilsTest, TestDFRSimilarityFactory, QueryResultKeyTest, AlternateDirectoryTest, TestDistributedGrouping, ShardRoutingCustomTest, SuggesterWFSTTest, TestOmitPositions, DirectSolrConnectionTest, TestCoreDiscovery, TestDistributedSearch, TestCollapseQParserPlugin, XsltUpdateRequestHandlerTest, AliasIntegrationTest, TestSolrDeletionPolicy2, TestIBSimilarityFactory, HdfsSyncSliceTest, TestFastWriter, BadIndexSchemaTest, HardAutoCommitTest, TestSweetSpotSimilarityFactory, TestQuerySenderListener, TestFoldingMultitermQuery, SolrIndexConfigTest, DirectSolrSpellCheckerTest, PathHierarchyTokenizerFactoryTest, RequiredFieldsTest, UniqFieldsUpdateProcessorFactoryTest, FieldAnalysisRequestHandlerTest, TestAddFieldRealTimeGet, TestStressUserVersions, TestAnalyzedSuggestions, HighlighterTest, LeaderElectionIntegrationTest, TestDefaultSimilarityFactory, TestRecoveryHdfs, TestFastOutputStream, DocumentAnalysisRequestHandlerTest, InfoHandlerTest, TestUpdate, TimeZoneUtilsTest, TestFastLRUCache, TestIndexingPerformance, TestCopyFieldCollectionResource, ShardSplitTest, SolrXmlInZkTest, PluginInfoTest, JSONWriterTest, MinimalSchemaTest, TestReloadAndDeleteDocs, TestArbitraryIndexDir, TestIndexSearcher, TestStressLucene, TestJmxIntegration, TestCodecSupport, SpellingQueryConverterTest, TestDistribDocBasedVersion, TestCSVResponseWriter, ChaosMonkeySafeLeaderTest, NumericFieldsTest, DistanceFunctionTest, SolrCmdDistributorTest, BadCopyFieldTest, CurrencyFieldXmlFileTest, PingRequestHandlerTest, TestSolr4Spatial, DocValuesMissingTest, RequestHandlersTest, OutputWriterTest, TestAnalyzeInfixSuggestions, ConvertedLegacyTest, TestPhraseSuggestions, TestSolrJ, DateFieldTest, CollectionsAPIDistributedZkTest, TestComponentsName, SuggesterTSTTest, TestDocBasedVersionConstraints, TestZkChroot, OpenCloseCoreStressTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, ZkCLITest, SliceStateUpdateTest, AutoCommitTest, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, TestStressReorder, TestMultiCoreConfBootstrap, TestReload, TestRangeQuery, SolrCoreTest, StatsComponentTest, SpellCheckComponentTest, TestFiltering, ShowFileRequestHandlerTest, SolrIndexSplitterTest, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, SolrRequestParserTest, NoCacheHeaderTest, SchemaVersionSpecificBehaviorTest, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, QueryEqualityTest, StatelessScriptUpdateProcessorFactoryTest, SolrInfoMBeanTest, CacheHeaderTest, TestSurroundQueryParser, DisMaxRequestHandlerTest, TestReversedWildcardFilterFactory, TermsComponentTest, TestValueSourceCache, TestSolrQueryParser, LoggingHandlerTest, IndexSchemaTest, JsonLoaderTest, CSVRequestHandlerTest, CoreContainerCoreInitFailuresTest, TestLFUCache, UpdateParamsTest, TestQuerySenderNoQuery, BadComponentTest, TestMergePolicyConfig, TestDocSet, TestBinaryField, TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, NotRequiredUniqueKeyTest, TestLuceneMatchVersion, SpellPossibilityIteratorTest, SynonymTokenizerTest, TestXIncludeConfig, TestNumberUtils, LegacyHTMLStripCharFilterTest, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, DateMathParserTest, ChaosMonkeyNothingIsSafeTest]
[junit4] Completed on J1 in 52.63s, 1 test, 1 failure <<< FAILURES!
[...truncated 192 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:401: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:471: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1259: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:902: There were test failures: 332 suites, 1464 tests, 1 failure, 35 ignored (6 assumptions)
Total time: 43 minutes 21 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_45 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0-ea-b109) - Build
# 8161 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/8161/
Java: 32bit/jdk1.8.0-ea-b109 -server -XX:+UseSerialGC
All tests passed
Build Log:
[...truncated 32280 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:421: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:360: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:66: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:135: The following files are missing svn:eol-style (or binary svn:mime-type):
* ./lucene/demo/src/java/org/apache/lucene/demo/facet/ExpressionAggregationFacetsExample.java
* ./lucene/demo/src/test/org/apache/lucene/demo/facet/TestExpressionAggregationFacetsExample.java
* ./lucene/facet/src/java/org/apache/lucene/facet/search/SumValueSourceFacetRequest.java
* ./lucene/facet/src/test/org/apache/lucene/facet/search/TestSumValueSourceFacetRequest.java
Total time: 48 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.8.0-ea-b109 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure