You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2013/06/16 08:31:09 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0-ea-b93) - Build # 6158 - Failure!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6158/
Java: 64bit/jdk1.8.0-ea-b93 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
shard3 is not consistent.  Got 96 from http://127.0.0.1:39572/_bbe/y/collection1lastClient and got 76 from http://127.0.0.1:44436/_bbe/y/collection1

Stack Trace:
java.lang.AssertionError: shard3 is not consistent.  Got 96 from http://127.0.0.1:39572/_bbe/y/collection1lastClient and got 76 from http://127.0.0.1:44436/_bbe/y/collection1
	at __randomizedtesting.SeedInfo.seed([A2AD1D4D9C35FB9E:234B9355EB6A9BA2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:491)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 9444 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4]   2> 71266 T287 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_bbe/y
[junit4:junit4]   2> 71276 T287 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1371363189183
[junit4:junit4]   2> 71279 T287 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 71281 T288 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 71380 T287 oasc.ZkTestServer.run start zk server on port:50502
[junit4:junit4]   2> 71431 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 71521 T294 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f271fe3 name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 71522 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 71523 T287 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 71554 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 71556 T296 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@44436347 name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 71557 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 71558 T287 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 71563 T287 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 71566 T287 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 71570 T287 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 71575 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 71576 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 71585 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 71585 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 71590 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 71591 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 71596 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 71596 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 71600 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 71600 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 71606 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 71607 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 71611 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 71612 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 71616 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 71616 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 71620 T287 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 71621 T287 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 71705 T287 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 71723 T287 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40980
[junit4:junit4]   2> 71724 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 71724 T287 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 71724 T287 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535
[junit4:junit4]   2> 71725 T287 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535/solr.xml
[junit4:junit4]   2> 71725 T287 oasc.CoreContainer.<init> New CoreContainer 1254812210
[junit4:junit4]   2> 71725 T287 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535/'
[junit4:junit4]   2> 71725 T287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535/'
[junit4:junit4]   2> 71782 T287 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 71782 T287 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 71782 T287 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 71782 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 71783 T287 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 71783 T287 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 71783 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 71783 T287 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 71783 T287 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 71784 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 71787 T287 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 71788 T287 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50502/solr
[junit4:junit4]   2> 71794 T287 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 71795 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 71797 T307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cfff4c2 name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 71798 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 71801 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 71808 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 71811 T309 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4310c520 name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 71811 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 71815 T287 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 71823 T287 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 71832 T287 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 71836 T287 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40980__bbe%2Fy
[junit4:junit4]   2> 71841 T287 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40980__bbe%2Fy
[junit4:junit4]   2> 71849 T287 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 71869 T287 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 71875 T287 oasc.Overseer.start Overseer (id=89873657971146755-127.0.0.1:40980__bbe%2Fy-n_0000000000) starting
[junit4:junit4]   2> 71893 T287 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 71909 T311 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 71910 T287 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 71916 T287 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 71922 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 71938 T310 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 71946 T312 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 71946 T312 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 73445 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 73446 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:40980/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:40980__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 73446 T310 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 73453 T310 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 73464 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 73949 T312 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535/collection1
[junit4:junit4]   2> 73950 T312 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 73951 T312 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 73951 T312 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 73953 T312 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535/collection1/'
[junit4:junit4]   2> 73953 T312 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535/collection1/lib/README' to classloader
[junit4:junit4]   2> 73954 T312 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 73988 T312 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 74018 T312 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 74020 T312 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 74024 T312 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 74359 T312 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 74366 T312 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 74368 T312 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 74391 T312 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 74396 T312 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 74400 T312 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 74402 T312 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 74402 T312 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 74403 T312 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 74404 T312 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 74404 T312 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 74405 T312 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 74413 T312 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1371363189535/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/control/data/
[junit4:junit4]   2> 74414 T312 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7dc415f3
[junit4:junit4]   2> 74414 T312 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 74414 T312 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/control/data
[junit4:junit4]   2> 74415 T312 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/control/data/index/
[junit4:junit4]   2> 74415 T312 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 74415 T312 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/control/data/index
[junit4:junit4]   2> 74520 T312 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 74521 T312 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 74523 T312 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 74523 T312 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 74524 T312 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 74525 T312 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 74525 T312 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 74526 T312 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 74526 T312 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 74527 T312 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 74527 T312 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 74530 T312 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 74535 T312 oass.SolrIndexSearcher.<init> Opening Searcher@5fd5e404 main
[junit4:junit4]   2> 74535 T312 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 74535 T312 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 74538 T313 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5fd5e404 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 74539 T312 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 74539 T312 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40980/_bbe/y collection:control_collection shard:shard1
[junit4:junit4]   2> 74543 T312 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 74550 T312 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 74558 T312 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 74561 T312 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 74561 T312 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 74563 T312 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40980/_bbe/y/collection1/
[junit4:junit4]   2> 74563 T312 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 74563 T312 oasc.SyncStrategy.syncToMe http://127.0.0.1:40980/_bbe/y/collection1/ has no replicas
[junit4:junit4]   2> 74563 T312 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40980/_bbe/y/collection1/
[junit4:junit4]   2> 74563 T312 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 74970 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75088 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 75122 T312 oasc.ZkController.register We are http://127.0.0.1:40980/_bbe/y/collection1/ and leader is http://127.0.0.1:40980/_bbe/y/collection1/
[junit4:junit4]   2> 75122 T312 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40980/_bbe/y
[junit4:junit4]   2> 75122 T312 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 75123 T312 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 75123 T312 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 75125 T312 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 75126 T287 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 75126 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 75127 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 75131 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 75133 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 75135 T316 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7354fb98 name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 75136 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 75137 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 75142 T287 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 75221 T287 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 75224 T287 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49250
[junit4:junit4]   2> 75225 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 75225 T287 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 75225 T287 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049
[junit4:junit4]   2> 75232 T287 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049/solr.xml
[junit4:junit4]   2> 75233 T287 oasc.CoreContainer.<init> New CoreContainer 895417231
[junit4:junit4]   2> 75233 T287 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049/'
[junit4:junit4]   2> 75233 T287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049/'
[junit4:junit4]   2> 75293 T287 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 75294 T287 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 75294 T287 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 75294 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 75294 T287 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 75295 T287 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 75295 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 75295 T287 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 75295 T287 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 75295 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 75299 T287 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 75299 T287 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50502/solr
[junit4:junit4]   2> 75300 T287 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 75300 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 75303 T327 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d3134ec name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 75304 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 75307 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 75312 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 75314 T329 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@872771c name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 75314 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 75319 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 76323 T287 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49250__bbe%2Fy
[junit4:junit4]   2> 76325 T287 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49250__bbe%2Fy
[junit4:junit4]   2> 76328 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 76329 T316 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 76329 T329 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 76330 T309 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 76335 T330 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 76336 T330 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 76596 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 76597 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:40980__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:40980/_bbe/y",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:40980__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 76605 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49250/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49250__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 76605 T310 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4]   2> 76606 T310 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 76659 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 76659 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 76659 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 77337 T330 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049/collection1
[junit4:junit4]   2> 77338 T330 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 77339 T330 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 77340 T330 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 77343 T330 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049/collection1/'
[junit4:junit4]   2> 77344 T330 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049/collection1/lib/README' to classloader
[junit4:junit4]   2> 77345 T330 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 77407 T330 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 77442 T330 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 77456 T330 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 77465 T330 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 77842 T330 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 77849 T330 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 77852 T330 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 77874 T330 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 77882 T330 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 77889 T330 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 77892 T330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 77893 T330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 77893 T330 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 77895 T330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 77895 T330 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 77895 T330 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 77896 T330 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1371363193049/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1/
[junit4:junit4]   2> 77896 T330 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7dc415f3
[junit4:junit4]   2> 77896 T330 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 77897 T330 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1
[junit4:junit4]   2> 77897 T330 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1/index/
[junit4:junit4]   2> 77898 T330 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 77898 T330 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1/index
[junit4:junit4]   2> 77902 T330 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 77902 T330 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 77904 T330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 77905 T330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 77905 T330 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 77906 T330 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 77906 T330 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 77907 T330 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 77907 T330 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 77908 T330 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 77908 T330 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 77910 T330 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 77915 T330 oass.SolrIndexSearcher.<init> Opening Searcher@3ab86177 main
[junit4:junit4]   2> 77915 T330 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 77916 T330 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 77920 T331 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3ab86177 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 77922 T330 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 77922 T330 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49250/_bbe/y collection:collection1 shard:shard1
[junit4:junit4]   2> 77923 T330 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 77933 T330 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 77936 T330 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 77936 T330 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 77936 T330 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49250/_bbe/y/collection1/
[junit4:junit4]   2> 77937 T330 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 77937 T330 oasc.SyncStrategy.syncToMe http://127.0.0.1:49250/_bbe/y/collection1/ has no replicas
[junit4:junit4]   2> 77937 T330 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49250/_bbe/y/collection1/
[junit4:junit4]   2> 77937 T330 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 78170 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78189 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 78189 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 78189 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 78199 T330 oasc.ZkController.register We are http://127.0.0.1:49250/_bbe/y/collection1/ and leader is http://127.0.0.1:49250/_bbe/y/collection1/
[junit4:junit4]   2> 78199 T330 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49250/_bbe/y
[junit4:junit4]   2> 78200 T330 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 78200 T330 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 78200 T330 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 78203 T330 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 78205 T287 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 78206 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 78207 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 78304 T287 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 78306 T287 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59684
[junit4:junit4]   2> 78307 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 78307 T287 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 78307 T287 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121
[junit4:junit4]   2> 78307 T287 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121/solr.xml
[junit4:junit4]   2> 78308 T287 oasc.CoreContainer.<init> New CoreContainer 100362174
[junit4:junit4]   2> 78308 T287 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121/'
[junit4:junit4]   2> 78308 T287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121/'
[junit4:junit4]   2> 78346 T287 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 78347 T287 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 78347 T287 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 78347 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 78347 T287 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 78348 T287 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 78348 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 78348 T287 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 78348 T287 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 78348 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 78352 T287 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 78353 T287 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50502/solr
[junit4:junit4]   2> 78353 T287 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 78353 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 78355 T343 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5f5e3c62 name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 78356 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 78358 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 78362 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 78364 T345 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2aee1509 name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 78364 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 78368 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 79373 T287 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59684__bbe%2Fy
[junit4:junit4]   2> 79375 T287 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59684__bbe%2Fy
[junit4:junit4]   2> 79379 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 79381 T316 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 79382 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 79382 T329 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 79382 T345 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 79383 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 79388 T309 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 79395 T346 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 79395 T346 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 79701 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 79702 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49250__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49250/_bbe/y",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49250__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 79707 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:59684/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:59684__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 79708 T310 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 79708 T310 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4]   2> 79740 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 79740 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 79740 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 79740 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 80396 T346 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121/collection1
[junit4:junit4]   2> 80397 T346 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 80399 T346 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 80399 T346 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 80402 T346 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121/collection1/'
[junit4:junit4]   2> 80403 T346 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121/collection1/lib/README' to classloader
[junit4:junit4]   2> 80403 T346 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 80459 T346 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 80495 T346 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 80497 T346 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 80503 T346 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 80858 T346 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 80864 T346 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 80866 T346 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 80896 T346 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 80900 T346 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 80905 T346 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 80907 T346 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 80908 T346 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 80909 T346 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 80911 T346 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 80911 T346 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 80911 T346 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 80911 T346 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1371363196121/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2/
[junit4:junit4]   2> 80911 T346 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7dc415f3
[junit4:junit4]   2> 80912 T346 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 80912 T346 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2
[junit4:junit4]   2> 80912 T346 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2/index/
[junit4:junit4]   2> 80913 T346 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 80913 T346 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2/index
[junit4:junit4]   2> 80917 T346 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 80917 T346 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 80919 T346 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 80919 T346 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 80920 T346 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 80921 T346 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 80921 T346 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 80921 T346 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 80922 T346 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 80922 T346 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 80922 T346 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 80924 T346 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 80927 T346 oass.SolrIndexSearcher.<init> Opening Searcher@4a35db7c main
[junit4:junit4]   2> 80927 T346 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 80927 T346 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 80935 T347 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a35db7c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 80937 T346 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 80937 T346 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59684/_bbe/y collection:collection1 shard:shard2
[junit4:junit4]   2> 80938 T346 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 80946 T346 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 80951 T346 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 80951 T346 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 80951 T346 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59684/_bbe/y/collection1/
[junit4:junit4]   2> 80951 T346 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 80951 T346 oasc.SyncStrategy.syncToMe http://127.0.0.1:59684/_bbe/y/collection1/ has no replicas
[junit4:junit4]   2> 80952 T346 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59684/_bbe/y/collection1/
[junit4:junit4]   2> 80952 T346 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 81255 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81271 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 81271 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 81271 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 81271 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 81316 T346 oasc.ZkController.register We are http://127.0.0.1:59684/_bbe/y/collection1/ and leader is http://127.0.0.1:59684/_bbe/y/collection1/
[junit4:junit4]   2> 81316 T346 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59684/_bbe/y
[junit4:junit4]   2> 81316 T346 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 81317 T346 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 81317 T346 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 81319 T346 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 81320 T287 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 81320 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 81320 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 81412 T287 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 81415 T287 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58708
[junit4:junit4]   2> 81415 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 81415 T287 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 81416 T287 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234
[junit4:junit4]   2> 81416 T287 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234/solr.xml
[junit4:junit4]   2> 81416 T287 oasc.CoreContainer.<init> New CoreContainer 1506510351
[junit4:junit4]   2> 81416 T287 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234/'
[junit4:junit4]   2> 81417 T287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234/'
[junit4:junit4]   2> 81456 T287 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 81457 T287 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 81457 T287 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 81457 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 81457 T287 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 81458 T287 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 81458 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 81458 T287 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 81458 T287 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 81459 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 81462 T287 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 81463 T287 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50502/solr
[junit4:junit4]   2> 81463 T287 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 81464 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 81466 T359 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45c02a03 name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 81466 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 81469 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 81474 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 81476 T361 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1635e688 name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 81476 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 81480 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 82484 T287 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58708__bbe%2Fy
[junit4:junit4]   2> 82487 T287 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58708__bbe%2Fy
[junit4:junit4]   2> 82490 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 82490 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 82491 T316 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 82491 T345 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 82491 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 82491 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 82491 T361 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 82493 T329 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 82493 T309 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 82498 T362 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 82498 T362 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 82784 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 82785 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:59684__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:59684/_bbe/y",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:59684__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 82790 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:58708/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:58708__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 82790 T310 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 82790 T310 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4]   2> 82819 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 82820 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 82820 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 82820 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 82819 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 83500 T362 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234/collection1
[junit4:junit4]   2> 83500 T362 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 83501 T362 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 83502 T362 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 83503 T362 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234/collection1/'
[junit4:junit4]   2> 83504 T362 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234/collection1/lib/README' to classloader
[junit4:junit4]   2> 83504 T362 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 83535 T362 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 83572 T362 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 83574 T362 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 83579 T362 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 83953 T362 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 83959 T362 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 83961 T362 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 83979 T362 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 83982 T362 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 83985 T362 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 83986 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 83987 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 83987 T362 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 83988 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 83988 T362 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 83989 T362 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 83989 T362 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1371363199234/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3/
[junit4:junit4]   2> 83989 T362 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7dc415f3
[junit4:junit4]   2> 83990 T362 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 83990 T362 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3
[junit4:junit4]   2> 83990 T362 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3/index/
[junit4:junit4]   2> 83991 T362 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 83991 T362 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3/index
[junit4:junit4]   2> 83995 T362 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 83995 T362 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 83998 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 83998 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 83999 T362 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 84000 T362 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 84000 T362 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 84000 T362 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 84001 T362 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 84001 T362 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 84002 T362 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 84004 T362 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 84009 T362 oass.SolrIndexSearcher.<init> Opening Searcher@3a71d0a9 main
[junit4:junit4]   2> 84010 T362 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 84010 T362 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 84014 T363 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a71d0a9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 84015 T362 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 84015 T362 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58708/_bbe/y collection:collection1 shard:shard3
[junit4:junit4]   2> 84016 T362 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 84024 T362 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 84026 T362 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 84027 T362 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 84027 T362 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58708/_bbe/y/collection1/
[junit4:junit4]   2> 84027 T362 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 84027 T362 oasc.SyncStrategy.syncToMe http://127.0.0.1:58708/_bbe/y/collection1/ has no replicas
[junit4:junit4]   2> 84027 T362 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58708/_bbe/y/collection1/
[junit4:junit4]   2> 84027 T362 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4]   2> 84330 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84340 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84341 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84340 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84340 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84340 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84386 T362 oasc.ZkController.register We are http://127.0.0.1:58708/_bbe/y/collection1/ and leader is http://127.0.0.1:58708/_bbe/y/collection1/
[junit4:junit4]   2> 84386 T362 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58708/_bbe/y
[junit4:junit4]   2> 84386 T362 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 84386 T362 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 84386 T362 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 84388 T362 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 84389 T287 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 84390 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 84390 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 84472 T287 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 84474 T287 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43491
[junit4:junit4]   2> 84475 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 84475 T287 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 84476 T287 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301
[junit4:junit4]   2> 84476 T287 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301/solr.xml
[junit4:junit4]   2> 84476 T287 oasc.CoreContainer.<init> New CoreContainer 1499189043
[junit4:junit4]   2> 84476 T287 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301/'
[junit4:junit4]   2> 84477 T287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301/'
[junit4:junit4]   2> 84517 T287 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 84517 T287 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 84517 T287 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 84518 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 84518 T287 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 84518 T287 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 84518 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 84519 T287 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 84519 T287 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 84519 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 84523 T287 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 84523 T287 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50502/solr
[junit4:junit4]   2> 84524 T287 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 84525 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 84528 T375 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@592e4272 name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 84528 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 84534 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 84538 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 84540 T377 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41e3d9dd name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 84540 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 84545 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 85549 T287 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43491__bbe%2Fy
[junit4:junit4]   2> 85551 T287 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43491__bbe%2Fy
[junit4:junit4]   2> 85555 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 85555 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 85556 T361 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 85556 T329 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 85556 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 85556 T377 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 85556 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 85557 T345 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 85557 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 85558 T309 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 85559 T316 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 85563 T378 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 85563 T378 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 85849 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 85850 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:58708__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:58708/_bbe/y",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard":"shard3",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:58708__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 85854 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43491/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:43491__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 85854 T310 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 85854 T310 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 85930 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 85930 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 85930 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 85930 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 85930 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 85930 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 86565 T378 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301/collection1
[junit4:junit4]   2> 86565 T378 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 86566 T378 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 86567 T378 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 86568 T378 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301/collection1/'
[junit4:junit4]   2> 86569 T378 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301/collection1/lib/README' to classloader
[junit4:junit4]   2> 86569 T378 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 86616 T378 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 86668 T378 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 86670 T378 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 86679 T378 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 87068 T378 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 87075 T378 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 87077 T378 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 87098 T378 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 87101 T378 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 87106 T378 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 87107 T378 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 87108 T378 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 87108 T378 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 87111 T378 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 87111 T378 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 87112 T378 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 87112 T378 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1371363202301/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/
[junit4:junit4]   2> 87112 T378 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7dc415f3
[junit4:junit4]   2> 87113 T378 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 87113 T378 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4
[junit4:junit4]   2> 87114 T378 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index/
[junit4:junit4]   2> 87114 T378 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 87114 T378 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index
[junit4:junit4]   2> 87120 T378 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 87120 T378 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 87122 T378 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 87122 T378 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 87122 T378 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 87123 T378 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 87123 T378 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 87123 T378 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 87124 T378 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 87124 T378 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 87124 T378 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 87126 T378 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 87129 T378 oass.SolrIndexSearcher.<init> Opening Searcher@b8e5298 main
[junit4:junit4]   2> 87129 T378 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 87129 T378 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 87133 T379 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@b8e5298 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 87135 T378 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 87135 T378 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43491/_bbe/y collection:collection1 shard:shard1
[junit4:junit4]   2> 87138 T378 oasc.ZkController.register We are http://127.0.0.1:43491/_bbe/y/collection1/ and leader is http://127.0.0.1:49250/_bbe/y/collection1/
[junit4:junit4]   2> 87139 T378 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43491/_bbe/y
[junit4:junit4]   2> 87139 T378 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 87139 T378 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C925 name=collection1 org.apache.solr.core.SolrCore@16889bbf url=http://127.0.0.1:43491/_bbe/y/collection1 node=127.0.0.1:43491__bbe%2Fy C925_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:43491/_bbe/y, state=down, node_name=127.0.0.1:43491__bbe%2Fy}
[junit4:junit4]   2> 87142 T380 C925 P43491 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 87143 T380 C925 P43491 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 87143 T378 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87143 T380 C925 P43491 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 87143 T380 C925 P43491 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 87144 T287 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 87144 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 87144 T380 C925 P43491 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 87145 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 87154 T325 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:43491__bbe%2Fy_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 87227 T287 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 87229 T287 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60408
[junit4:junit4]   2> 87230 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 87230 T287 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 87231 T287 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055
[junit4:junit4]   2> 87231 T287 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055/solr.xml
[junit4:junit4]   2> 87231 T287 oasc.CoreContainer.<init> New CoreContainer 548348227
[junit4:junit4]   2> 87231 T287 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055/'
[junit4:junit4]   2> 87232 T287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055/'
[junit4:junit4]   2> 87280 T287 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 87281 T287 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 87281 T287 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 87281 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 87282 T287 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 87282 T287 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 87282 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 87283 T287 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 87283 T287 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 87283 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 87289 T287 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 87290 T287 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50502/solr
[junit4:junit4]   2> 87290 T287 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 87291 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87294 T392 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c324b4e name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87295 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87297 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 87302 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 87304 T394 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45ad2988 name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 87304 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 87309 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 87443 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 87444 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:43491__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43491/_bbe/y",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:43491__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 87449 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 87449 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 87449 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 87449 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 87449 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 87449 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 87450 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 88155 T325 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:43491__bbe%2Fy_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 88155 T325 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:43491__bbe%252Fy&core=collection1&version=2&onlyIfLeader=true&coreNodeName=127.0.0.1:43491__bbe%252Fy_collection1&checkLive=true&action=PREPRECOVERY&wt=javabin&state=recovering} status=0 QTime=1002 
[junit4:junit4]   2> 88314 T287 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60408__bbe%2Fy
[junit4:junit4]   2> 88315 T287 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60408__bbe%2Fy
[junit4:junit4]   2> 88317 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 88318 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 88318 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 88318 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 88318 T329 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 88319 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88319 T316 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 88319 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88319 T361 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 88319 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88321 T377 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 88322 T345 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 88323 T394 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 88325 T309 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 88335 T395 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 88335 T395 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 88958 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 88960 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:60408/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:60408__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 88960 T310 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 88961 T310 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard2
[junit4:junit4]   2> 88970 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88970 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88970 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88971 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88970 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88970 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 88970 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 89336 T395 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055/collection1
[junit4:junit4]   2> 89337 T395 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 89338 T395 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 89339 T395 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 89341 T395 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055/collection1/'
[junit4:junit4]   2> 89342 T395 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055/collection1/lib/README' to classloader
[junit4:junit4]   2> 89343 T395 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 89384 T395 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 89420 T395 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 89422 T395 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 89427 T395 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 89787 T395 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 89793 T395 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 89795 T395 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 89813 T395 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 89818 T395 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 89823 T395 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 89825 T395 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 89825 T395 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 89825 T395 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 89827 T395 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 89827 T395 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 89827 T395 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 89827 T395 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1371363205055/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/
[junit4:junit4]   2> 89828 T395 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7dc415f3
[junit4:junit4]   2> 89828 T395 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 89829 T395 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5
[junit4:junit4]   2> 89829 T395 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index/
[junit4:junit4]   2> 89829 T395 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 89829 T395 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index
[junit4:junit4]   2> 89870 T395 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 89871 T395 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 89873 T395 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 89873 T395 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 89874 T395 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 89874 T395 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 89875 T395 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 89875 T395 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 89875 T395 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 89876 T395 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 89876 T395 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 89877 T395 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 89881 T395 oass.SolrIndexSearcher.<init> Opening Searcher@66d27888 main
[junit4:junit4]   2> 89882 T395 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 89882 T395 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 89888 T395 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 89888 T395 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60408/_bbe/y collection:collection1 shard:shard2
[junit4:junit4]   2> 89888 T396 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@66d27888 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 89891 T395 oasc.ZkController.register We are http://127.0.0.1:60408/_bbe/y/collection1/ and leader is http://127.0.0.1:59684/_bbe/y/collection1/
[junit4:junit4]   2> 89891 T395 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60408/_bbe/y
[junit4:junit4]   2> 89891 T395 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 89892 T395 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C926 name=collection1 org.apache.solr.core.SolrCore@1311d6da url=http://127.0.0.1:60408/_bbe/y/collection1 node=127.0.0.1:60408__bbe%2Fy C926_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:60408/_bbe/y, state=down, node_name=127.0.0.1:60408__bbe%2Fy}
[junit4:junit4]   2> 89892 T397 C926 P60408 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 89893 T397 C926 P60408 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 89893 T397 C926 P60408 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 89893 T397 C926 P60408 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 89893 T395 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C927 name=collection1 org.apache.solr.core.SolrCore@1311d6da url=http://127.0.0.1:60408/_bbe/y/collection1 node=127.0.0.1:60408__bbe%2Fy C927_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:60408/_bbe/y, state=down, node_name=127.0.0.1:60408__bbe%2Fy}
[junit4:junit4]   2> 89901 T397 C927 P60408 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 89902 T287 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 89903 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 89903 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 89907 T341 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:60408__bbe%2Fy_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 89986 T287 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 89989 T287 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39572
[junit4:junit4]   2> 89990 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 89990 T287 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 89990 T287 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814
[junit4:junit4]   2> 89990 T287 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814/solr.xml
[junit4:junit4]   2> 89991 T287 oasc.CoreContainer.<init> New CoreContainer 1927095441
[junit4:junit4]   2> 89991 T287 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814/'
[junit4:junit4]   2> 89991 T287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814/'
[junit4:junit4]   2> 90039 T287 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 90040 T287 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 90040 T287 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 90040 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 90041 T287 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 90041 T287 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 90041 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 90042 T287 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 90042 T287 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 90042 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 90048 T287 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 90049 T287 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50502/solr
[junit4:junit4]   2> 90049 T287 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 90051 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90054 T409 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@501d0e84 name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90054 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90057 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 90062 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 90065 T411 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@449e6add name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 90065 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 90073 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C928 name=collection1 org.apache.solr.core.SolrCore@16889bbf url=http://127.0.0.1:43491/_bbe/y/collection1 node=127.0.0.1:43491__bbe%2Fy C928_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:43491/_bbe/y, state=recovering, shard=shard1, node_name=127.0.0.1:43491__bbe%2Fy}
[junit4:junit4]   2> 90156 T380 C928 P43491 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:49250/_bbe/y/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 90160 T380 C928 P43491 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 90164 T380 C928 P43491 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:43491/_bbe/y START replicas=[http://127.0.0.1:49250/_bbe/y/collection1/] nUpdates=100
[junit4:junit4]   2> 90165 T380 C928 P43491 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 90165 T380 C928 P43491 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 90165 T380 C928 P43491 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 90165 T380 C928 P43491 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 90166 T380 C928 P43491 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 90166 T380 C928 P43491 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:49250/_bbe/y/collection1/. core=collection1
[junit4:junit4]   2> 90166 T380 C928 P43491 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C929 name=collection1 org.apache.solr.core.SolrCore@646fa29b url=http://127.0.0.1:49250/_bbe/y/collection1 node=127.0.0.1:49250__bbe%2Fy C929_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:49250/_bbe/y, state=active, shard=shard1, node_name=127.0.0.1:49250__bbe%2Fy, leader=true}
[junit4:junit4]   2> 90179 T324 C929 P49250 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/get params={getVersions=100&version=2&qt=/get&wt=javabin&distrib=false} status=0 QTime=0 
[junit4:junit4]   2> 90181 T323 C929 P49250 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 90183 T323 C929 P49250 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 90184 T323 C929 P49250 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 90189 T323 C929 P49250 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty1/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 90189 T323 C929 P49250 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 90190 T323 C929 P49250 oass.SolrIndexSearcher.<init> Opening Searcher@609ffe76 realtime
[junit4:junit4]   2> 90190 T323 C929 P49250 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 90190 T323 C929 P49250 oasup.LogUpdateProcessor.finish [collection1] webapp=/_bbe/y path=/update params={commit=true&openSearcher=false&version=2&commit_end_point=true&softCommit=false&wt=javabin&waitSearcher=true} {commit=} 0 10
[junit4:junit4]   2> ASYNC  NEW_CORE C930 name=collection1 org.apache.solr.core.SolrCore@16889bbf url=http://127.0.0.1:43491/_bbe/y/collection1 node=127.0.0.1:43491__bbe%2Fy C930_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:43491/_bbe/y, state=recovering, shard=shard1, node_name=127.0.0.1:43491__bbe%2Fy}
[junit4:junit4]   2> 90210 T380 C930 P43491 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 90211 T380 C930 P43491 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4]   2> 90216 T380 C930 P43491 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 90220 T322 C929 P49250 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 90221 T322 C929 P49250 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=indexversion&version=2&qt=/replication&wt=javabin} status=0 QTime=4 
[junit4:junit4]   2> 90222 T380 C930 P43491 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 90222 T380 C930 P43491 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 90222 T380 C930 P43491 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 90224 T324 C929 P49250 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=filelist&generation=2&version=2&qt=/replication&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 90225 T380 C930 P43491 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 90226 T380 C930 P43491 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index.20130616021328132
[junit4:junit4]   2> 90226 T380 C930 P43491 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index.20130616021328132 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7de11295 fullCopy=false
[junit4:junit4]   2> 90232 T323 C929 P49250 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=filecontent&generation=2&file=segments_2&qt=/replication&wt=filestream&checksum=true} status=0 QTime=2 
[junit4:junit4]   2> 90237 T380 C930 P43491 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 90247 T380 C930 P43491 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 90247 T380 C930 P43491 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 90252 T380 C930 P43491 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 90253 T380 C930 P43491 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 90253 T380 C930 P43491 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 90253 T380 C930 P43491 oass.SolrIndexSearcher.<init> Opening Searcher@4d104f86 main
[junit4:junit4]   2> 90254 T379 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4d104f86 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 90255 T380 C930 P43491 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index.20130616021328132 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index.20130616021328132;done=true>>]
[junit4:junit4]   2> 90255 T380 C930 P43491 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index.20130616021328132
[junit4:junit4]   2> 90255 T380 C930 P43491 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty4/index.20130616021328132
[junit4:junit4]   2> 90256 T380 C930 P43491 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 90256 T380 C930 P43491 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 90257 T380 C930 P43491 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 90257 T380 C930 P43491 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 90259 T380 C930 P43491 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 90480 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 90481 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:60408__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:60408/_bbe/y",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:60408__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 90485 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:43491__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:43491/_bbe/y",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:43491__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 90509 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 90510 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 90510 T411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 90509 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 90509 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 90509 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 90509 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 90510 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 90907 T341 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:60408__bbe%2Fy_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 90908 T341 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:60408__bbe%252Fy&core=collection1&version=2&onlyIfLeader=true&coreNodeName=127.0.0.1:60408__bbe%252Fy_collection1&checkLive=true&action=PREPRECOVERY&wt=javabin&state=recovering} status=0 QTime=1001 
[junit4:junit4]   2> 91078 T287 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39572__bbe%2Fy
[junit4:junit4]   2> 91079 T287 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39572__bbe%2Fy
[junit4:junit4]   2> 91081 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 91082 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 91082 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 91081 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 91081 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 91083 T411 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91083 T411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 91083 T345 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91083 T394 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91083 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 91083 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 91084 T316 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91085 T377 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91085 T361 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91085 T329 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91086 T309 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 91091 T414 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 91091 T414 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 92022 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 92023 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:39572/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:39572__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 92023 T310 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 92024 T310 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard3
[junit4:junit4]   2> 92030 T411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 92030 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 92030 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 92030 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 92030 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 92030 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 92030 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 92030 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 92093 T414 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814/collection1
[junit4:junit4]   2> 92093 T414 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 92094 T414 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 92094 T414 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 92095 T414 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814/collection1/'
[junit4:junit4]   2> 92096 T414 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814/collection1/lib/README' to classloader
[junit4:junit4]   2> 92096 T414 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 92125 T414 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 92160 T414 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 92162 T414 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 92166 T414 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 92512 T414 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 92518 T414 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 92520 T414 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 92540 T414 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 92545 T414 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 92548 T414 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 92550 T414 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 92550 T414 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 92551 T414 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 92553 T414 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 92553 T414 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 92553 T414 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 92553 T414 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1371363207814/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/
[junit4:junit4]   2> 92554 T414 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7dc415f3
[junit4:junit4]   2> 92554 T414 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 92555 T414 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6
[junit4:junit4]   2> 92555 T414 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index/
[junit4:junit4]   2> 92555 T414 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 92556 T414 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index
[junit4:junit4]   2> 92559 T414 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 92560 T414 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 92561 T414 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 92562 T414 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 92562 T414 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 92563 T414 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 92563 T414 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 92563 T414 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 92563 T414 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 92564 T414 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 92564 T414 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 92565 T414 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 92568 T414 oass.SolrIndexSearcher.<init> Opening Searcher@1621fa01 main
[junit4:junit4]   2> 92569 T414 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 92569 T414 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 92573 T415 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1621fa01 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 92575 T414 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 92575 T414 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39572/_bbe/y collection:collection1 shard:shard3
[junit4:junit4]   2> 92579 T414 oasc.ZkController.register We are http://127.0.0.1:39572/_bbe/y/collection1/ and leader is http://127.0.0.1:58708/_bbe/y/collection1/
[junit4:junit4]   2> 92579 T414 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39572/_bbe/y
[junit4:junit4]   2> 92579 T414 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 92579 T414 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C931 name=collection1 org.apache.solr.core.SolrCore@32a36fe8 url=http://127.0.0.1:39572/_bbe/y/collection1 node=127.0.0.1:39572__bbe%2Fy C931_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:39572/_bbe/y, state=down, node_name=127.0.0.1:39572__bbe%2Fy}
[junit4:junit4]   2> 92579 T416 C931 P39572 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 92580 T416 C931 P39572 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 92580 T416 C931 P39572 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 92580 T414 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 92580 T416 C931 P39572 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 92581 T287 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 92582 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 92582 T416 C931 P39572 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 92582 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 92587 T354 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: 127.0.0.1:39572__bbe%2Fy_collection1, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 92667 T287 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 92669 T287 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49681
[junit4:junit4]   2> 92670 T287 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 92670 T287 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 92670 T287 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371363210493
[junit4:junit4]   2> 92670 T287 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371363210493/solr.xml
[junit4:junit4]   2> 92671 T287 oasc.CoreContainer.<init> New CoreContainer 68844925
[junit4:junit4]   2> 92671 T287 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371363210493/'
[junit4:junit4]   2> 92671 T287 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1371363210493/'
[junit4:junit4]   2> 92721 T287 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 92722 T287 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 92722 T287 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 92723 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 92723 T287 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 92723 T287 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 92724 T287 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 92724 T287 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 92725 T287 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 92725 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 92731 T287 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 92732 T287 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50502/solr
[junit4:junit4]   2> 92732 T287 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 92733 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 92736 T428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@351332fc name:ZooKeeperConnection Watcher:127.0.0.1:50502 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 92737 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 92739 T287 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 92742 T287 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 92744 T430 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58aeda9 name:ZooKeeperConnection Watcher:127.0.0.1:50502/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 92744 T287 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 92748 T287 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C932 name=collection1 org.apache.solr.core.SolrCore@1311d6da url=http://127.0.0.1:60408/_bbe/y/collection1 node=127.0.0.1:60408__bbe%2Fy C932_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:60408/_bbe/y, state=recovering, shard=shard2, node_name=127.0.0.1:60408__bbe%2Fy}
[junit4:junit4]   2> 92910 T397 C932 P60408 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:59684/_bbe/y/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 92910 T397 C932 P60408 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:60408/_bbe/y START replicas=[http://127.0.0.1:59684/_bbe/y/collection1/] nUpdates=100
[junit4:junit4]   2> 92911 T397 C932 P60408 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 92912 T397 C932 P60408 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 92912 T397 C932 P60408 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 92912 T397 C932 P60408 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 92912 T397 C932 P60408 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 92913 T397 C932 P60408 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:59684/_bbe/y/collection1/. core=collection1
[junit4:junit4]   2> 92914 T397 C932 P60408 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C933 name=collection1 org.apache.solr.core.SolrCore@4934ef8d url=http://127.0.0.1:59684/_bbe/y/collection1 node=127.0.0.1:59684__bbe%2Fy C933_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:59684/_bbe/y, state=active, shard=shard2, node_name=127.0.0.1:59684__bbe%2Fy, leader=true}
[junit4:junit4]   2> 92921 T339 C933 P59684 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/get params={getVersions=100&version=2&qt=/get&wt=javabin&distrib=false} status=0 QTime=0 
[junit4:junit4]   2> 92922 T338 C933 P59684 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 92926 T338 C933 P59684 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 92927 T338 C933 P59684 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 92939 T338 C933 P59684 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty2/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 92940 T338 C933 P59684 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 92941 T338 C933 P59684 oass.SolrIndexSearcher.<init> Opening Searcher@20fce86e realtime
[junit4:junit4]   2> 92941 T338 C933 P59684 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 92941 T338 C933 P59684 oasup.LogUpdateProcessor.finish [collection1] webapp=/_bbe/y path=/update params={commit=true&openSearcher=false&version=2&commit_end_point=true&softCommit=false&wt=javabin&waitSearcher=true} {commit=} 0 19
[junit4:junit4]   2> 92943 T397 C932 P60408 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 92943 T397 C932 P60408 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 92947 T340 C933 P59684 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 92948 T340 C933 P59684 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=indexversion&version=2&qt=/replication&wt=javabin} status=0 QTime=1 
[junit4:junit4]   2> 92949 T397 C932 P60408 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 92949 T397 C932 P60408 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 92949 T397 C932 P60408 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 92952 T339 C933 P59684 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=filelist&generation=2&version=2&qt=/replication&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 92953 T397 C932 P60408 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 92954 T397 C932 P60408 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index.20130616021330861
[junit4:junit4]   2> 92955 T397 C932 P60408 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index.20130616021330861 lockFactory=org.apache.lucene.store.NativeFSLockFactory@487e4082 fullCopy=false
[junit4:junit4]   2> 92958 T338 C933 P59684 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=filecontent&generation=2&file=segments_2&qt=/replication&wt=filestream&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 92960 T397 C932 P60408 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 92970 T397 C932 P60408 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 92971 T397 C932 P60408 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 92974 T397 C932 P60408 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 92975 T397 C932 P60408 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 92975 T397 C932 P60408 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 92976 T397 C932 P60408 oass.SolrIndexSearcher.<init> Opening Searcher@6b4b31c8 main
[junit4:junit4]   2> 92977 T396 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6b4b31c8 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 92978 T397 C932 P60408 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index.20130616021330861 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index.20130616021330861;done=true>>]
[junit4:junit4]   2> 92979 T397 C932 P60408 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index.20130616021330861
[junit4:junit4]   2> 92979 T397 C932 P60408 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty5/index.20130616021330861
[junit4:junit4]   2> 92979 T397 C932 P60408 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 92980 T397 C932 P60408 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 92980 T397 C932 P60408 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 92980 T397 C932 P60408 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 92983 T397 C932 P60408 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 93540 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 93541 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:39572__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:39572/_bbe/y",
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "shard":"shard3",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:39572__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 93562 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:60408__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:60408/_bbe/y",
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "shard":"shard2",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:60408__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 93571 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93571 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93572 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93572 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93572 T430 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93572 T411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93571 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93571 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93571 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93588 T354 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: 127.0.0.1:39572__bbe%2Fy_collection1, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 93588 T354 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:39572__bbe%252Fy&core=collection1&version=2&onlyIfLeader=true&coreNodeName=127.0.0.1:39572__bbe%252Fy_collection1&checkLive=true&action=PREPRECOVERY&wt=javabin&state=recovering} status=0 QTime=1001 
[junit4:junit4]   2> 93752 T287 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49681__bbe%2Fy
[junit4:junit4]   2> 93754 T287 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49681__bbe%2Fy
[junit4:junit4]   2> 93757 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93757 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93758 T411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93757 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93757 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 93758 T430 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93758 T430 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 93758 T361 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93759 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 93759 T394 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93759 T377 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93759 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 93759 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 93760 T316 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93762 T309 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93762 T345 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93763 T411 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93764 T329 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 93774 T433 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 93774 T433 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 95087 T310 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 95088 T310 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":"3",
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49681/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49681__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 95089 T310 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 95089 T310 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard=shard1
[junit4:junit4]   2> 95096 T394 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 95096 T309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 95096 T345 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 95096 T430 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 95096 T316 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 95096 T329 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 95096 T361 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 95096 T377 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 95097 T411 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> ASYNC  NEW_CORE C934 name=collection1 org.apache.solr.core.SolrCore@32a36fe8 url=http://127.0.0.1:39572/_bbe/y/collection1 node=127.0.0.1:39572__bbe%2Fy C934_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:39572/_bbe/y, state=recovering, shard=shard3, node_name=127.0.0.1:39572__bbe%2Fy}
[junit4:junit4]   2> 95590 T416 C934 P39572 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:58708/_bbe/y/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 95590 T416 C934 P39572 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39572/_bbe/y START replicas=[http://127.0.0.1:58708/_bbe/y/collection1/] nUpdates=100
[junit4:junit4]   2> 95591 T416 C934 P39572 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 95591 T416 C934 P39572 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 95592 T416 C934 P39572 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 95592 T416 C934 P39572 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 95592 T416 C934 P39572 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 95592 T416 C934 P39572 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:58708/_bbe/y/collection1/. core=collection1
[junit4:junit4]   2> 95593 T416 C934 P39572 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C935 name=collection1 org.apache.solr.core.SolrCore@3f1255e2 url=http://127.0.0.1:58708/_bbe/y/collection1 node=127.0.0.1:58708__bbe%2Fy C935_STATE=coll:collection1 core:collection1 props:{collection=collection1, core=collection1, base_url=http://127.0.0.1:58708/_bbe/y, state=active, shard=shard3, node_name=127.0.0.1:58708__bbe%2Fy, leader=true}
[junit4:junit4]   2> 95602 T355 C935 P58708 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/get params={getVersions=100&version=2&qt=/get&wt=javabin&distrib=false} status=0 QTime=1 
[junit4:junit4]   2> 95603 T357 C935 P58708 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 95607 T357 C935 P58708 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 95608 T357 C935 P58708 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 95619 T357 C935 P58708 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty3/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 95620 T357 C935 P58708 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 95621 T357 C935 P58708 oass.SolrIndexSearcher.<init> Opening Searcher@5e233aca realtime
[junit4:junit4]   2> 95621 T357 C935 P58708 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 95621 T357 C935 P58708 oasup.LogUpdateProcessor.finish [collection1] webapp=/_bbe/y path=/update params={commit=true&openSearcher=false&version=2&commit_end_point=true&softCommit=false&wt=javabin&waitSearcher=true} {commit=} 0 18
[junit4:junit4]   2> 95623 T416 C934 P39572 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 95623 T416 C934 P39572 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 95626 T354 C935 P58708 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 95627 T354 C935 P58708 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=indexversion&version=2&qt=/replication&wt=javabin} status=0 QTime=1 
[junit4:junit4]   2> 95628 T416 C934 P39572 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 95628 T416 C934 P39572 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 95628 T416 C934 P39572 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 95631 T355 C935 P58708 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=filelist&generation=2&version=2&qt=/replication&wt=javabin} status=0 QTime=0 
[junit4:junit4]   2> 95632 T416 C934 P39572 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 95633 T416 C934 P39572 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index.20130616021333540
[junit4:junit4]   2> 95634 T416 C934 P39572 oash.SnapPuller.fetchLatestIndex Starting download to org.apache.lucene.store.MMapDirectory@/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index.20130616021333540 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2d56023f fullCopy=false
[junit4:junit4]   2> 95637 T357 C935 P58708 oasc.SolrCore.execute [collection1] webapp=/_bbe/y path=/replication params={command=filecontent&generation=2&file=segments_2&qt=/replication&wt=filestream&checksum=true} status=0 QTime=0 
[junit4:junit4]   2> 95639 T416 C934 P39572 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 95650 T416 C934 P39572 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 95650 T416 C934 P39572 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 95654 T416 C934 P39572 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=2
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 		commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index,segFN=segments_2,generation=2}
[junit4:junit4]   2> 95655 T416 C934 P39572 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 95655 T416 C934 P39572 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 95656 T416 C934 P39572 oass.SolrIndexSearcher.<init> Opening Searcher@21434ce main
[junit4:junit4]   2> 95657 T415 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@21434ce main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 95658 T416 C934 P39572 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index.20130616021333540 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index.20130616021333540;done=true>>]
[junit4:junit4]   2> 95659 T416 C934 P39572 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index.20130616021333540
[junit4:junit4]   2> 95659 T416 C934 P39572 oasc.CachingDirectoryFactory.closeCacheValue Removing directory before core close: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty6/index.20130616021333540
[junit4:junit4]   2> 95660 T416 C934 P39572 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 95660 T416 C934 P39572 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - register

[...truncated too long message...]

hildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 295451 T917 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 295452 T917 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 295453 T917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:34237__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:34237/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:34237__bbe%2Fy"}
[junit4:junit4]   2> 295456 T917 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:34237__bbe%2Fy_collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "numShards":null,
[junit4:junit4]   2> 	  "shard_range":null,
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:34237/_bbe/y",
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:34237__bbe%2Fy",
[junit4:junit4]   2> 	  "shard_state":"active"}
[junit4:junit4]   2> 295459 T744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 295461 T287 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_bbe/y,null}
[junit4:junit4]   2> 295512 T287 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 41462
[junit4:junit4]   2> 295513 T287 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=493806295
[junit4:junit4]   2> 295653 T662 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 295653 T662 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 296514 T287 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 296515 T287 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 296517 T287 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4c9a9458
[junit4:junit4]   2> 296518 T287 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=26,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=172,cumulative_deletesById=90,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 296518 T287 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 296519 T287 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 296519 T287 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 296520 T287 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 296521 T287 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4:junit4]   2> 296521 T287 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty11/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty11/index;done=false>>]
[junit4:junit4]   2> 296522 T287 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty11/index
[junit4:junit4]   2> 296522 T287 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty11 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty11;done=false>>]
[junit4:junit4]   2> 296522 T287 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1371363189183/jetty11
[junit4:junit4]   2> 296523 T917 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=89873657971146801-127.0.0.1:41462__bbe%2Fy-n_0000000026) am no longer a leader.
[junit4:junit4]   2> 296525 T744 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 296525 T744 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 296526 T744 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 296547 T287 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_bbe/y,null}
[junit4:junit4]   2> 296599 T287 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 38860
[junit4:junit4]   2> 296647 T287 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 296649 T287 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:50502 50502
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   1> safe stop:
[junit4:junit4]   1> safe stop:
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=A2AD1D4D9C35FB9E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=cs_CZ -Dtests.timezone=US/East-Indiana -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE  225s J1 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: shard3 is not consistent.  Got 96 from http://127.0.0.1:39572/_bbe/y/collection1lastClient and got 76 from http://127.0.0.1:44436/_bbe/y/collection1
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([A2AD1D4D9C35FB9E:234B9355EB6A9BA2]:0)
[junit4:junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
[junit4:junit4]    > 	at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:137)
[junit4:junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:815)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 296659 T287 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 225400 T286 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 296739 T706 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 296739 T706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 296739 T706 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=cs_CZ, timezone=US/East-Indiana
[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.8.0-ea (64-bit)/cpus=8,threads=2,free=62863464,total=162643968
[junit4:junit4]   2> NOTE: All tests run in this JVM: [DistanceFunctionTest, TestFiltering, TestJmxMonitoredMap, TestSolr4Spatial, TestSolrXMLSerializer, TestMergePolicyConfig, DirectSolrSpellCheckerTest, TermVectorComponentDistributedTest, TestPropInject, TestStressReorder, CopyFieldTest, MultiTermTest, TestUtils, SortByFunctionTest, TestCSVResponseWriter, TestAtomicUpdateErrorCases, ChaosMonkeySafeLeaderTest]
[junit4:junit4] Completed on J1 in 225.93s, 1 test, 1 failure <<< FAILURES!

[...truncated 659 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:437: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 296 suites, 1243 tests, 1 failure, 16 ignored (7 assumptions)

Total time: 37 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0-ea-b93 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure



Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_21) - Build # 6159 - Still Failing!

Posted by Simon Willnauer <si...@gmail.com>.
thanks for looking in to this :)

simon


On Mon, Jun 17, 2013 at 8:25 AM, Smiley, David W. <ds...@mitre.org> wrote:

> Reported the bug with a narrow test case:
> https://issues.apache.org/jira/browse/LUCENE-5062
> I'm currently looking into a fix.  In the mean time the existing tests
> should very, very rarely hit this.  The other failures were on 4x, which
> failed because of a former problem in the test that was fixed in trunk
> that I forgot to back-port to 4x until tonight.
>
> ~ David
>
> On 6/16/13 3:40 AM, "Policeman Jenkins Server" <je...@thetaphi.de>
> wrote:
>
> >Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6159/
> >Java: 32bit/jdk1.7.0_21 -server -XX:+UseG1GC
> >
> >1 tests failed.
> >FAILED:
> >org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testCont
> >ains {#5 seed=[594050BB28CB401B:ED310C391B43B6BA]}
> >
> >Error Message:
> >Should have matched I
> >#0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) ,
> >Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0))
> >Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)
> >
> >Stack Trace:
> >java.lang.AssertionError: Should have matched I
> >#0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) ,
> >Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0))
> >Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)
> >       at
> >__randomizedtesting.SeedInfo.seed([594050BB28CB401B:ED310C391B43B6BA]:0)
> >       at org.junit.Assert.fail(Assert.java:93)
> >       at
> >org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(Spa
> >tialOpRecursivePrefixTreeTest.java:289)
> >       at
> >org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(S
> >patialOpRecursivePrefixTreeTest.java:282)
> >       at
> >org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testCont
> >ains(SpatialOpRecursivePrefixTreeTest.java:103)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> >57)
> >       at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
> >pl.java:43)
> >       at java.lang.reflect.Method.invoke(Method.java:601)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunne
> >r.java:1559)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedR
> >unner.java:79)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedR
> >unner.java:737)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedR
> >unner.java:773)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedR
> >unner.java:787)
> >       at
> >org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSet
> >upTeardownChained.java:50)
> >       at
> >org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCa
> >cheSanity.java:51)
> >       at
> >org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAf
> >terRule.java:46)
> >       at
> >com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.e
> >valuate(SystemPropertiesInvariantRule.java:55)
> >       at
> >org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThread
> >AndTestName.java:49)
> >       at
> >org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleI
> >gnoreAfterMaxFailures.java:70)
> >       at
> >org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.
> >java:48)
> >       at
> >com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> >ntAdapter.java:36)
> >       at
> >com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(T
> >hreadLeakControl.java:358)
> >       at
> >com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(Th
> >readLeakControl.java:782)
> >       at
> >com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeak
> >Control.java:442)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Randomiz
> >edRunner.java:746)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedR
> >unner.java:648)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedR
> >unner.java:682)
> >       at
> >com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedR
> >unner.java:693)
> >       at
> >org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAf
> >terRule.java:46)
> >       at
> >org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClas
> >sName.java:42)
> >       at
> >com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.e
> >valuate(SystemPropertiesInvariantRule.java:55)
> >       at
> >com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRu
> >le$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >       at
> >com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRu
> >le$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> >       at
> >com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> >ntAdapter.java:36)
> >       at
> >org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAsser
> >tionsRequired.java:43)
> >       at
> >org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.
> >java:48)
> >       at
> >org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleI
> >gnoreAfterMaxFailures.java:70)
> >       at
> >org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreT
> >estSuites.java:55)
> >       at
> >com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
> >ntAdapter.java:36)
> >       at
> >com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(T
> >hreadLeakControl.java:358)
> >       at java.lang.Thread.run(Thread.java:722)
> >
> >
> >
> >
> >Build Log:
> >[...truncated 8309 lines...]
> >[junit4:junit4] Suite:
> >org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
> >evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
> >evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
> >Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
> >Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
> >evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
> >evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
> >evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
> >Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
> >evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
> >evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
> >Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
> >evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
> >evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
> >Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
> >evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
> >evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
> >evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
> >evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
> >Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
> >evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
> >evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
> >evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
> >Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
> >evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
> >evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
> >Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
> >Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
> >evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   2>
> >Ig:ShapePair(Rect(minX=60.0,maxX=68.0,minY=-124.0,maxY=112.0) ,
> >Rect(minX=8.0,maxX=62.0,minY=-114.0,maxY=128.0))
> >Qg:Rect(minX=48.0,maxX=66.0,minY=38.0,maxY=68.0)
> >[junit4:junit4]   2> NOTE: reproduce with: ant test
> >-Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method="testContains
> >{#5 seed=[594050BB28CB401B:ED310C391B43B6BA]}"
> >-Dtests.seed=594050BB28CB401B -Dtests.multiplier=3 -Dtests.slow=true
> >-Dtests.locale=es_MX -Dtests.timezone=Israel
> >-Dtests.file.encoding=US-ASCII
> >[junit4:junit4] FAILURE 0.10s J1 |
> >SpatialOpRecursivePrefixTreeTest.testContains {#5
> >seed=[594050BB28CB401B:ED310C391B43B6BA]} <<<
> >[junit4:junit4]    > Throwable #1: java.lang.AssertionError: Should have
> >matched I #0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) ,
> >Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0))
> >Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)
> >[junit4:junit4]    >   at
> >__randomizedtesting.SeedInfo.seed([594050BB28CB401B:ED310C391B43B6BA]:0)
> >[junit4:junit4]    >   at
> >org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(Spa
> >tialOpRecursivePrefixTreeTest.java:289)
> >[junit4:junit4]    >   at
> >org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(S
> >patialOpRecursivePrefixTreeTest.java:282)
> >[junit4:junit4]    >   at
> >org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testCont
> >ains(SpatialOpRecursivePrefixTreeTest.java:103)
> >[junit4:junit4]    >   at java.lang.Thread.run(Thread.java:722)
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
> >evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
> >evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
> >Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
> >Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
> >Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
> >evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
> >Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
> >evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
> >Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
> >evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
> >Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
> >evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
> >evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   1> Strategy:
> >RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
> >Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
> >worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
> >[junit4:junit4]   2> NOTE: test params are: codec=Asserting,
> >sim=DefaultSimilarity, locale=es_MX, timezone=Israel
> >[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation
> >1.7.0_21 (32-bit)/cpus=8,threads=1,free=10780240,total=68157440
> >[junit4:junit4]   2> NOTE: All tests run in this JVM:
> >[SpatialArgsParserTest, TestTestFramework,
> >TestTermQueryPrefixGridStrategy, TestPointVectorStrategy,
> >TestBBoxStrategy, SpatialOpRecursivePrefixTreeTest]
> >[junit4:junit4] Completed on J1 in 3.53s, 42 tests, 1 failure <<<
> >FAILURES!
> >
> >[...truncated 10 lines...]
> >BUILD FAILED
> >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The
> >following error occurred while executing this line:
> >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The
> >following error occurred while executing this line:
> >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The
> >following error occurred while executing this line:
> >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:557:
> >The following error occurred while executing this line:
> >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
> >:1827: The following error occurred while executing this line:
> >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml
> >:61: The following error occurred while executing this line:
> >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
> >:1243: The following error occurred while executing this line:
> >/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
> >:887: There were test failures: 13 suites, 74 tests, 1 failure, 2 ignored
> >(2 assumptions)
> >
> >Total time: 17 minutes 49 seconds
> >Build step 'Invoke Ant' marked build as failure
> >Description set: Java: 32bit/jdk1.7.0_21 -server -XX:+UseG1GC
> >Archiving artifacts
> >Recording test results
> >Email was triggered for: Failure
> >Sending email for trigger: Failure
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_21) - Build # 6159 - Still Failing!

Posted by "Smiley, David W." <ds...@mitre.org>.
Reported the bug with a narrow test case:
https://issues.apache.org/jira/browse/LUCENE-5062
I'm currently looking into a fix.  In the mean time the existing tests
should very, very rarely hit this.  The other failures were on 4x, which
failed because of a former problem in the test that was fixed in trunk
that I forgot to back-port to 4x until tonight.

~ David

On 6/16/13 3:40 AM, "Policeman Jenkins Server" <je...@thetaphi.de> wrote:

>Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6159/
>Java: 32bit/jdk1.7.0_21 -server -XX:+UseG1GC
>
>1 tests failed.
>FAILED:  
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testCont
>ains {#5 seed=[594050BB28CB401B:ED310C391B43B6BA]}
>
>Error Message:
>Should have matched I
>#0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) ,
>Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0))
>Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)
>
>Stack Trace:
>java.lang.AssertionError: Should have matched I
>#0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) ,
>Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0))
>Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)
>	at 
>__randomizedtesting.SeedInfo.seed([594050BB28CB401B:ED310C391B43B6BA]:0)
>	at org.junit.Assert.fail(Assert.java:93)
>	at 
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(Spa
>tialOpRecursivePrefixTreeTest.java:289)
>	at 
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(S
>patialOpRecursivePrefixTreeTest.java:282)
>	at 
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testCont
>ains(SpatialOpRecursivePrefixTreeTest.java:103)
>	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>	at 
>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
>57)
>	at 
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm
>pl.java:43)
>	at java.lang.reflect.Method.invoke(Method.java:601)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunne
>r.java:1559)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedR
>unner.java:79)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedR
>unner.java:737)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedR
>unner.java:773)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedR
>unner.java:787)
>	at 
>org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSet
>upTeardownChained.java:50)
>	at 
>org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCa
>cheSanity.java:51)
>	at 
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAf
>terRule.java:46)
>	at 
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.e
>valuate(SystemPropertiesInvariantRule.java:55)
>	at 
>org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThread
>AndTestName.java:49)
>	at 
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleI
>gnoreAfterMaxFailures.java:70)
>	at 
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.
>java:48)
>	at 
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>ntAdapter.java:36)
>	at 
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(T
>hreadLeakControl.java:358)
>	at 
>com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(Th
>readLeakControl.java:782)
>	at 
>com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeak
>Control.java:442)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Randomiz
>edRunner.java:746)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedR
>unner.java:648)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedR
>unner.java:682)
>	at 
>com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedR
>unner.java:693)
>	at 
>org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAf
>terRule.java:46)
>	at 
>org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClas
>sName.java:42)
>	at 
>com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.e
>valuate(SystemPropertiesInvariantRule.java:55)
>	at 
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRu
>le$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at 
>com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRu
>le$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
>	at 
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>ntAdapter.java:36)
>	at 
>org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAsser
>tionsRequired.java:43)
>	at 
>org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.
>java:48)
>	at 
>org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleI
>gnoreAfterMaxFailures.java:70)
>	at 
>org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreT
>estSuites.java:55)
>	at 
>com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(Stateme
>ntAdapter.java:36)
>	at 
>com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(T
>hreadLeakControl.java:358)
>	at java.lang.Thread.run(Thread.java:722)
>
>
>
>
>Build Log:
>[...truncated 8309 lines...]
>[junit4:junit4] Suite:
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
>evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
>Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
>evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
>Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
>evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
>evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
>Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
>evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
>Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
>evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
>evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   2>
>Ig:ShapePair(Rect(minX=60.0,maxX=68.0,minY=-124.0,maxY=112.0) ,
>Rect(minX=8.0,maxX=62.0,minY=-114.0,maxY=128.0))
>Qg:Rect(minX=48.0,maxX=66.0,minY=38.0,maxY=68.0)
>[junit4:junit4]   2> NOTE: reproduce with: ant test
>-Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method="testContains
>{#5 seed=[594050BB28CB401B:ED310C391B43B6BA]}"
>-Dtests.seed=594050BB28CB401B -Dtests.multiplier=3 -Dtests.slow=true
>-Dtests.locale=es_MX -Dtests.timezone=Israel
>-Dtests.file.encoding=US-ASCII
>[junit4:junit4] FAILURE 0.10s J1 |
>SpatialOpRecursivePrefixTreeTest.testContains {#5
>seed=[594050BB28CB401B:ED310C391B43B6BA]} <<<
>[junit4:junit4]    > Throwable #1: java.lang.AssertionError: Should have
>matched I #0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) ,
>Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0))
>Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)
>[junit4:junit4]    > 	at
>__randomizedtesting.SeedInfo.seed([594050BB28CB401B:ED310C391B43B6BA]:0)
>[junit4:junit4]    > 	at
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(Spa
>tialOpRecursivePrefixTreeTest.java:289)
>[junit4:junit4]    > 	at
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(S
>patialOpRecursivePrefixTreeTest.java:282)
>[junit4:junit4]    > 	at
>org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testCont
>ains(SpatialOpRecursivePrefixTreeTest.java:103)
>[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxL
>evels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
>evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
>Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
>Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
>Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxL
>evels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(max
>Levels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(max
>Levels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxL
>evels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxL
>evels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxL
>evels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   1> Strategy:
>RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(max
>Levels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc,
>worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
>[junit4:junit4]   2> NOTE: test params are: codec=Asserting,
>sim=DefaultSimilarity, locale=es_MX, timezone=Israel
>[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation
>1.7.0_21 (32-bit)/cpus=8,threads=1,free=10780240,total=68157440
>[junit4:junit4]   2> NOTE: All tests run in this JVM:
>[SpatialArgsParserTest, TestTestFramework,
>TestTermQueryPrefixGridStrategy, TestPointVectorStrategy,
>TestBBoxStrategy, SpatialOpRecursivePrefixTreeTest]
>[junit4:junit4] Completed on J1 in 3.53s, 42 tests, 1 failure <<<
>FAILURES!
>
>[...truncated 10 lines...]
>BUILD FAILED
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The
>following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The
>following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The
>following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:557:
>The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
>:1827: The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml
>:61: The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
>:1243: The following error occurred while executing this line:
>/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml
>:887: There were test failures: 13 suites, 74 tests, 1 failure, 2 ignored
>(2 assumptions)
>
>Total time: 17 minutes 49 seconds
>Build step 'Invoke Ant' marked build as failure
>Description set: Java: 32bit/jdk1.7.0_21 -server -XX:+UseG1GC
>Archiving artifacts
>Recording test results
>Email was triggered for: Failure
>Sending email for trigger: Failure
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_21) - Build # 6159 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6159/
Java: 32bit/jdk1.7.0_21 -server -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains {#5 seed=[594050BB28CB401B:ED310C391B43B6BA]}

Error Message:
Should have matched I #0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) , Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0)) Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)

Stack Trace:
java.lang.AssertionError: Should have matched I #0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) , Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0)) Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)
	at __randomizedtesting.SeedInfo.seed([594050BB28CB401B:ED310C391B43B6BA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:289)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:282)
	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains(SpatialOpRecursivePrefixTreeTest.java:103)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 8309 lines...]
[junit4:junit4] Suite: org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   2> Ig:ShapePair(Rect(minX=60.0,maxX=68.0,minY=-124.0,maxY=112.0) , Rect(minX=8.0,maxX=62.0,minY=-114.0,maxY=128.0)) Qg:Rect(minX=48.0,maxX=66.0,minY=38.0,maxY=68.0)
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SpatialOpRecursivePrefixTreeTest -Dtests.method="testContains {#5 seed=[594050BB28CB401B:ED310C391B43B6BA]}" -Dtests.seed=594050BB28CB401B -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_MX -Dtests.timezone=Israel -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE 0.10s J1 | SpatialOpRecursivePrefixTreeTest.testContains {#5 seed=[594050BB28CB401B:ED310C391B43B6BA]} <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: Should have matched I #0:ShapePair(Rect(minX=61.0,maxX=67.0,minY=-122.0,maxY=110.0) , Rect(minX=10.0,maxX=61.0,minY=-113.0,maxY=126.0)) Q:Rect(minX=50.0,maxX=64.0,minY=39.0,maxY=66.0)
[junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([594050BB28CB401B:ED310C391B43B6BA]:0)
[junit4:junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.fail(SpatialOpRecursivePrefixTreeTest.java:289)
[junit4:junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.doTest(SpatialOpRecursivePrefixTreeTest.java:282)
[junit4:junit4]    > 	at org.apache.lucene.spatial.prefix.SpatialOpRecursivePrefixTreeTest.testContains(SpatialOpRecursivePrefixTreeTest.java:103)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:4,SPG:(QuadPrefixTree(maxLevels:8,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:2,SPG:(QuadPrefixTree(maxLevels:6,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-1,SPG:(QuadPrefixTree(maxLevels:3,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-3,SPG:(QuadPrefixTree(maxLevels:1,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:1,SPG:(QuadPrefixTree(maxLevels:5,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:0,SPG:(QuadPrefixTree(maxLevels:4,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:3,SPG:(QuadPrefixTree(maxLevels:7,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   1> Strategy: RecursivePrefixTreeStrategy(prefixGridScanLevel:-2,SPG:(QuadPrefixTree(maxLevels:2,ctx:SpatialContext{geo=false, calculator=CartesianDistCalc, worldBounds=Rect(minX=0.0,maxX=256.0,minY=-128.0,maxY=128.0)})))
[junit4:junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=es_MX, timezone=Israel
[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_21 (32-bit)/cpus=8,threads=1,free=10780240,total=68157440
[junit4:junit4]   2> NOTE: All tests run in this JVM: [SpatialArgsParserTest, TestTestFramework, TestTermQueryPrefixGridStrategy, TestPointVectorStrategy, TestBBoxStrategy, SpatialOpRecursivePrefixTreeTest]
[junit4:junit4] Completed on J1 in 3.53s, 42 tests, 1 failure <<< FAILURES!

[...truncated 10 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:557: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1827: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1243: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:887: There were test failures: 13 suites, 74 tests, 1 failure, 2 ignored (2 assumptions)

Total time: 17 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_21 -server -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure