You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/07/08 14:31:24 UTC
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_60) - Build #
10757 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10757/
Java: 32bit/jdk1.7.0_60 -server -XX:+UseSerialGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 473 from http://127.0.0.1:39389/collection1lastClient and got 459 from http://127.0.0.1:40137/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 473 from http://127.0.0.1:39389/collection1lastClient and got 459 from http://127.0.0.1:40137/collection1
at __randomizedtesting.SeedInfo.seed([6352A7B455F6A249:E2B429AC22A9C275]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1132)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1111)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11515 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/init-core-data-001
[junit4] 2> 741469 T2789 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 741469 T2789 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 741472 T2789 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 741473 T2789 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 741473 T2790 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 741573 T2789 oasc.ZkTestServer.run start zk server on port:58093
[junit4] 2> 741574 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 741576 T2796 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8732f0 name:ZooKeeperConnection Watcher:127.0.0.1:58093 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 741576 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 741576 T2789 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 741579 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 741580 T2798 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c7bd8b name:ZooKeeperConnection Watcher:127.0.0.1:58093/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 741581 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 741581 T2789 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 741583 T2789 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 741584 T2789 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 741586 T2789 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 741587 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 741587 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 741590 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 741590 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 741592 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 741593 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 741594 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 741594 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 741596 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 741596 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 741598 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 741598 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 741600 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 741600 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 741602 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 741602 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 741604 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 741604 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 741606 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 741606 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 741608 T2789 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 741608 T2789 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 741823 T2789 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 741826 T2789 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54452
[junit4] 2> 741827 T2789 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 741828 T2789 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 741828 T2789 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002
[junit4] 2> 741828 T2789 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002/'
[junit4] 2> 741845 T2789 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002/solr.xml
[junit4] 2> 741871 T2789 oasc.CoreContainer.<init> New CoreContainer 16854990
[junit4] 2> 741871 T2789 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002/]
[junit4] 2> 741872 T2789 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 741872 T2789 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 741873 T2789 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 741873 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 741873 T2789 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 741874 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 741874 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 741874 T2789 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 741874 T2789 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 741875 T2789 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 741875 T2789 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 741876 T2789 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 741876 T2789 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58093/solr
[junit4] 2> 741884 T2789 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 741885 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 741886 T2809 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a6a1a4 name:ZooKeeperConnection Watcher:127.0.0.1:58093 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 741886 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 741888 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 741889 T2811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d60d90 name:ZooKeeperConnection Watcher:127.0.0.1:58093/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 741889 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 741890 T2789 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 741892 T2789 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 741895 T2789 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 741897 T2789 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 741900 T2789 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 741902 T2789 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 741903 T2789 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54452_
[junit4] 2> 741904 T2789 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54452_
[junit4] 2> 741906 T2789 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 741908 T2789 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 741910 T2789 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:54452_
[junit4] 2> 741911 T2789 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 741912 T2789 oasc.Overseer.start Overseer (id=92066391381377027-127.0.0.1:54452_-n_0000000000) starting
[junit4] 2> 741915 T2789 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 741922 T2813 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 741922 T2789 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 741924 T2789 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 741925 T2789 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 741927 T2812 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 741930 T2814 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 741930 T2814 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 741931 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 741931 T2814 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 741932 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 741933 T2812 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:54452",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:54452_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 741933 T2812 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 741933 T2812 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 741935 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 741935 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 742931 T2814 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 742932 T2814 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 742933 T2814 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 742933 T2814 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 742934 T2814 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 742934 T2814 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002/collection1/'
[junit4] 2> 742935 T2814 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 742935 T2814 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 742935 T2814 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 742959 T2814 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 742981 T2814 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 742982 T2814 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 742987 T2814 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 743159 T2814 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 743160 T2814 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 743161 T2814 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 743164 T2814 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 743180 T2814 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 743180 T2814 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 743181 T2814 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/control/data/
[junit4] 2> 743181 T2814 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9d5c12
[junit4] 2> 743181 T2814 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/control/data
[junit4] 2> 743182 T2814 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/control/data/index/
[junit4] 2> 743182 T2814 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 743182 T2814 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/control/data/index
[junit4] 2> 743183 T2814 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 743188 T2814 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 743189 T2814 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 743192 T2814 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 743192 T2814 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 743192 T2814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 743192 T2814 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 743192 T2814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 743193 T2814 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 743193 T2814 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 743193 T2814 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 743193 T2814 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 743194 T2814 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 743194 T2814 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 743194 T2814 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 743194 T2814 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 743195 T2814 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 743195 T2814 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 743195 T2814 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 743196 T2814 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 743198 T2814 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 743199 T2814 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 743200 T2814 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 743200 T2814 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 743201 T2814 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 743201 T2814 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 743202 T2814 oass.SolrIndexSearcher.<init> Opening Searcher@18e2f24[collection1] main
[junit4] 2> 743202 T2814 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 743203 T2814 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 743203 T2814 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 743203 T2814 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 743203 T2814 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 743203 T2814 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 743204 T2814 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 743204 T2814 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 743205 T2814 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 743206 T2814 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 743206 T2814 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 743207 T2814 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 743208 T2815 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18e2f24[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 743210 T2814 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 743211 T2818 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:54452 collection:control_collection shard:shard1
[junit4] 2> 743211 T2789 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 743211 T2789 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 743211 T2818 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 743213 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 743214 T2820 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@69e9fe name:ZooKeeperConnection Watcher:127.0.0.1:58093/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 743214 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 743215 T2789 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 743216 T2818 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 743217 T2789 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 743217 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743218 T2818 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 743218 T2818 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1414 name=collection1 org.apache.solr.core.SolrCore@7b0a9e url=http://127.0.0.1:54452/collection1 node=127.0.0.1:54452_ C1414_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:54452, core=collection1, node_name=127.0.0.1:54452_}
[junit4] 2> 743218 T2818 C1414 P54452 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:54452/collection1/
[junit4] 2> 743218 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 743218 T2818 C1414 P54452 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 743219 T2818 C1414 P54452 oasc.SyncStrategy.syncToMe http://127.0.0.1:54452/collection1/ has no replicas
[junit4] 2> 743219 T2818 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:54452/collection1/ shard1
[junit4] 2> 743219 T2818 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 743220 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743223 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743225 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743326 T2820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 743326 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 743374 T2818 oasc.ZkController.register We are http://127.0.0.1:54452/collection1/ and leader is http://127.0.0.1:54452/collection1/
[junit4] 2> 743375 T2818 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54452
[junit4] 2> 743375 T2818 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 743375 T2818 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 743375 T2818 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 743376 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743376 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743376 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743377 T2818 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 743378 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 743378 T2812 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:54452",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:54452_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 743380 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 743446 T2789 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 743447 T2789 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 743449 T2789 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36255
[junit4] 2> 743450 T2789 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 743450 T2789 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 743450 T2789 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003
[junit4] 2> 743451 T2789 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003/'
[junit4] 2> 743461 T2789 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003/solr.xml
[junit4] 2> 743482 T2820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 743482 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 743485 T2789 oasc.CoreContainer.<init> New CoreContainer 26468524
[junit4] 2> 743485 T2789 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003/]
[junit4] 2> 743486 T2789 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 743486 T2789 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 743486 T2789 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 743487 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 743487 T2789 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 743487 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 743487 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 743488 T2789 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 743488 T2789 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 743489 T2789 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 743489 T2789 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 743489 T2789 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 743490 T2789 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58093/solr
[junit4] 2> 743490 T2789 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 743491 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 743492 T2831 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1834745 name:ZooKeeperConnection Watcher:127.0.0.1:58093 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 743492 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 743494 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 743495 T2833 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15864cd name:ZooKeeperConnection Watcher:127.0.0.1:58093/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 743495 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 743499 T2789 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 744501 T2789 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36255_
[junit4] 2> 744502 T2789 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36255_
[junit4] 2> 744509 T2834 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 744509 T2834 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 744509 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744509 T2834 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 744510 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744510 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744511 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 744511 T2812 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:36255",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:36255_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 744511 T2812 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 744512 T2812 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 744513 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 744615 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744615 T2820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 744615 T2833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 745510 T2834 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 745510 T2834 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 745511 T2834 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 745511 T2834 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 745512 T2834 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 745512 T2834 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003/collection1/'
[junit4] 2> 745512 T2834 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 745513 T2834 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 745513 T2834 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 745537 T2834 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 745568 T2834 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 745570 T2834 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 745575 T2834 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 745750 T2834 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 745751 T2834 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 745752 T2834 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 745755 T2834 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 745772 T2834 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 745772 T2834 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 745772 T2834 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty1/
[junit4] 2> 745772 T2834 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9d5c12
[junit4] 2> 745773 T2834 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty1
[junit4] 2> 745773 T2834 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty1/index/
[junit4] 2> 745773 T2834 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 745774 T2834 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty1/index
[junit4] 2> 745774 T2834 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 745777 T2834 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 745777 T2834 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 745781 T2834 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 745781 T2834 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 745782 T2834 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 745782 T2834 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 745782 T2834 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 745782 T2834 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 745783 T2834 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 745783 T2834 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 745783 T2834 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 745783 T2834 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 745784 T2834 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 745784 T2834 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 745784 T2834 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 745784 T2834 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 745785 T2834 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 745785 T2834 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 745786 T2834 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 745788 T2834 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 745789 T2834 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 745790 T2834 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 745790 T2834 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 745791 T2834 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 745791 T2834 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 745792 T2834 oass.SolrIndexSearcher.<init> Opening Searcher@119a661[collection1] main
[junit4] 2> 745792 T2834 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 745793 T2834 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 745793 T2834 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 745793 T2834 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 745793 T2834 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 745794 T2834 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 745794 T2834 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 745795 T2834 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 745795 T2834 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 745795 T2834 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 745798 T2835 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@119a661[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 745800 T2834 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 745800 T2838 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36255 collection:collection1 shard:shard1
[junit4] 2> 745801 T2789 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 745801 T2838 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 745801 T2789 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 745805 T2838 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 745806 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745807 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745807 T2838 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 745807 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745807 T2838 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 745808 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C1415 name=collection1 org.apache.solr.core.SolrCore@c4043d url=http://127.0.0.1:36255/collection1 node=127.0.0.1:36255_ C1415_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:36255, core=collection1, node_name=127.0.0.1:36255_}
[junit4] 2> 745808 T2838 C1415 P36255 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36255/collection1/
[junit4] 2> 745809 T2838 C1415 P36255 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 745809 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745810 T2838 C1415 P36255 oasc.SyncStrategy.syncToMe http://127.0.0.1:36255/collection1/ has no replicas
[junit4] 2> 745810 T2838 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36255/collection1/ shard1
[junit4] 2> 745810 T2838 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 745813 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745815 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745917 T2833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 745917 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 745917 T2820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 745965 T2838 oasc.ZkController.register We are http://127.0.0.1:36255/collection1/ and leader is http://127.0.0.1:36255/collection1/
[junit4] 2> 745965 T2838 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36255
[junit4] 2> 745965 T2838 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 745966 T2838 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 745966 T2838 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 745967 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745967 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745967 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 745967 T2838 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 745968 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 745969 T2812 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:36255",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:36255_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 745971 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 746027 T2789 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 746028 T2789 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 746029 T2789 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43923
[junit4] 2> 746030 T2789 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 746030 T2789 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 746030 T2789 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004
[junit4] 2> 746031 T2789 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004/'
[junit4] 2> 746041 T2789 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004/solr.xml
[junit4] 2> 746063 T2789 oasc.CoreContainer.<init> New CoreContainer 14317721
[junit4] 2> 746063 T2789 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004/]
[junit4] 2> 746064 T2789 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 746064 T2789 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 746065 T2789 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 746065 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 746065 T2789 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 746065 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 746066 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 746066 T2789 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 746066 T2789 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 746067 T2789 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 746067 T2789 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 746067 T2789 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 746068 T2789 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58093/solr
[junit4] 2> 746068 T2789 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 746069 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 746070 T2849 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@123a405 name:ZooKeeperConnection Watcher:127.0.0.1:58093 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 746070 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 746072 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 746072 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 746072 T2833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 746072 T2820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 746073 T2851 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17dc019 name:ZooKeeperConnection Watcher:127.0.0.1:58093/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 746074 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 746078 T2789 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 747080 T2789 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43923_
[junit4] 2> 747081 T2789 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43923_
[junit4] 2> 747089 T2852 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 747090 T2852 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 747090 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 747090 T2852 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 747091 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 747091 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 747092 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 747093 T2812 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:43923",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43923_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 747093 T2812 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 747093 T2812 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 747095 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 747197 T2833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 747197 T2851 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 747197 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 747197 T2820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748091 T2852 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 748091 T2852 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 748092 T2852 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 748092 T2852 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 748093 T2852 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 748093 T2852 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004/collection1/'
[junit4] 2> 748094 T2852 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 748094 T2852 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 748094 T2852 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 748130 T2852 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 748151 T2852 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 748153 T2852 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 748158 T2852 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 748338 T2852 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 748340 T2852 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 748341 T2852 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 748344 T2852 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 748359 T2852 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 748359 T2852 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 748360 T2852 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty2/
[junit4] 2> 748360 T2852 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@9d5c12
[junit4] 2> 748360 T2852 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty2
[junit4] 2> 748361 T2852 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty2/index/
[junit4] 2> 748361 T2852 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 748361 T2852 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty2/index
[junit4] 2> 748362 T2852 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=26, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 748365 T2852 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 748366 T2852 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 748369 T2852 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 748369 T2852 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 748370 T2852 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 748370 T2852 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 748370 T2852 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 748370 T2852 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 748371 T2852 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 748371 T2852 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 748371 T2852 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 748371 T2852 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 748372 T2852 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 748372 T2852 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 748372 T2852 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 748372 T2852 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 748373 T2852 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 748373 T2852 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 748374 T2852 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 748376 T2852 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 748377 T2852 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 748378 T2852 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 748378 T2852 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=26, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
[junit4] 2> 748379 T2852 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 748379 T2852 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 748380 T2852 oass.SolrIndexSearcher.<init> Opening Searcher@658448[collection1] main
[junit4] 2> 748380 T2852 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 748381 T2852 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 748381 T2852 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 748381 T2852 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 748381 T2852 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 748381 T2852 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 748382 T2852 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 748382 T2852 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 748383 T2852 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 748383 T2852 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 748385 T2853 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@658448[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 748387 T2852 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 748388 T2856 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43923 collection:collection1 shard:shard3
[junit4] 2> 748388 T2789 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4] 2> 748388 T2789 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 748388 T2856 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 748392 T2856 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 748393 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748394 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748394 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748394 T2856 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 748394 T2856 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C1416 name=collection1 org.apache.solr.core.SolrCore@4472f0 url=http://127.0.0.1:43923/collection1 node=127.0.0.1:43923_ C1416_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:43923, core=collection1, node_name=127.0.0.1:43923_}
[junit4] 2> 748395 T2856 C1416 P43923 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43923/collection1/
[junit4] 2> 748395 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 748395 T2856 C1416 P43923 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 748395 T2856 C1416 P43923 oasc.SyncStrategy.syncToMe http://127.0.0.1:43923/collection1/ has no replicas
[junit4] 2> 748395 T2856 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43923/collection1/ shard3
[junit4] 2> 748395 T2856 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 748397 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748399 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748401 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748502 T2851 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748502 T2833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748502 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748502 T2820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748550 T2856 oasc.ZkController.register We are http://127.0.0.1:43923/collection1/ and leader is http://127.0.0.1:43923/collection1/
[junit4] 2> 748550 T2856 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43923
[junit4] 2> 748550 T2856 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 748550 T2856 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 748551 T2856 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 748552 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748552 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748552 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748552 T2856 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 748553 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 748554 T2812 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:43923",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:43923_",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 748555 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 748619 T2789 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 748619 T2789 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 748621 T2789 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:46793
[junit4] 2> 748622 T2789 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 748622 T2789 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 748623 T2789 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-005
[junit4] 2> 748623 T2789 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-005/'
[junit4] 2> 748634 T2789 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-005/solr.xml
[junit4] 2> 748657 T2833 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748657 T2789 oasc.CoreContainer.<init> New CoreContainer 6302704
[junit4] 2> 748658 T2789 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-005/]
[junit4] 2> 748657 T2820 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748657 T2851 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748657 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 748659 T2789 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 748660 T2789 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 748660 T2789 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 748660 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 748660 T2789 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 748661 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 748661 T2789 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 748661 T2789 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 748662 T2789 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 748662 T2789 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 748662 T2789 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 748663 T2789 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 748663 T2789 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:58093/solr
[junit4] 2> 748663 T2789 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 748664 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 748665 T2867 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11c9934 name:ZooKeeperConnection Watcher:127.0.0.1:58093 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 748666 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 748671 T2789 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 748671 T2869 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c14c46 name:ZooKeeperConnection Watcher:127.0.0.1:58093/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 748672 T2789 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 748676 T2789 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 749679 T2789 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46793_
[junit4] 2> 749680 T2789 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46793_
[junit4] 2> 749688 T2870 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 749688 T2870 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 749689 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 749689 T2870 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 749689 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 749690 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 749690 T2812 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 749691 T2812 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:46793",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:46793_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 749691 T2812 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 749691 T2812 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 749693 T2811 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 749795 T2811 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.jso
[...truncated too long message...]
State.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 926666 T2789 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 926667 T3773 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 926677 T2789 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 926678 T2789 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 926678 T2789 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty12/index.20140708081551854 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty12/index.20140708081551854;done=false>>]
[junit4] 2> 926679 T2789 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty12/index.20140708081551854
[junit4] 2> 926679 T2789 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty12;done=false>>]
[junit4] 2> 926679 T2789 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001/tempDir-001/jetty12
[junit4] 2> 926680 T3959 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:40137_
[junit4] 2> 926704 T2789 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 926756 T2789 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> 926757 T2789 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58093 58093
[junit4] 2> 927517 T2790 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:58093 58093
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.method=testDistribSearch -Dtests.seed=6352A7B455F6A249 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=da_DK -Dtests.timezone=America/Marigot -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 186s J0 | ChaosMonkeySafeLeaderTest.testDistribSearch <<<
[junit4] > Throwable #1: java.lang.AssertionError: shard2 is not consistent. Got 473 from http://127.0.0.1:39389/collection1lastClient and got 459 from http://127.0.0.1:40137/collection1
[junit4] > at __randomizedtesting.SeedInfo.seed([6352A7B455F6A249:E2B429AC22A9C275]:0)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1132)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1111)
[junit4] > at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 927525 T2789 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-6352A7B455F6A249-001
[junit4] 2> 186060 T2788 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 5 leaked thread(s).
[junit4] 2> 927982 T3868 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 928088 T3792 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 928183 T3961 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 928183 T3773 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: test params are: codec=Lucene42, sim=DefaultSimilarity, locale=da_DK, timezone=America/Marigot
[junit4] 2> NOTE: Linux 3.8.0-42-generic i386/Oracle Corporation 1.7.0_60 (32-bit)/cpus=8,threads=1,free=32422928,total=441712640
[junit4] 2> NOTE: All tests run in this JVM: [HdfsLockFactoryTest, TestFastLRUCache, TestLeaderElectionZkExpiry, AnalyticsQueryTest, BadCopyFieldTest, ZkNodePropsTest, FunctionTest, TestAddFieldRealTimeGet, PrimUtilsTest, CoreAdminRequestStatusTest, TestConfig, MBeansHandlerTest, AnalysisErrorHandlingTest, SpellPossibilityIteratorTest, HdfsSyncSliceTest, TestBinaryResponseWriter, TestSerializedLuceneMatchVersion, TestSearchPerf, OverseerCollectionProcessorTest, HttpPartitionTest, TestComponentsName, TestArbitraryIndexDir, DocExpirationUpdateProcessorFactoryTest, TestSolrIndexConfig, ResponseLogComponentTest, RequiredFieldsTest, PathHierarchyTokenizerFactoryTest, ResponseHeaderTest, TestDFRSimilarityFactory, TestInfoStreamLogging, PingRequestHandlerTest, TestFastOutputStream, PreAnalyzedFieldTest, TestReRankQParserPlugin, TestModifyConfFiles, CollectionsAPIAsyncDistributedZkTest, StatelessScriptUpdateProcessorFactoryTest, TestSolrDeletionPolicy2, ReplicationFactorTest, SignatureUpdateProcessorFactoryTest, TestManagedResource, AddSchemaFieldsUpdateProcessorFactoryTest, TestCloudManagedSchema, TestCloudManagedSchemaConcurrent, SolrIndexSplitterTest, TestRandomDVFaceting, DocumentBuilderTest, FieldFacetExtrasTest, TestOrdValues, TestClassNameShortening, TestManagedSchema, TestCustomSort, TestSolrDeletionPolicy1, SuggesterWFSTTest, ExpressionTest, EchoParamsTest, BadIndexSchemaTest, HighlighterMaxOffsetTest, RegexBoostProcessorTest, FieldMutatingUpdateProcessorTest, TestAnalyzeInfixSuggestions, DocumentAnalysisRequestHandlerTest, QueryEqualityTest, TestCodecSupport, TestFoldingMultitermQuery, RAMDirectoryFactoryTest, TestJmxIntegration, ReturnFieldsTest, TestAnalyzedSuggestions, AbstractAnalyticsStatsTest, TestComplexPhraseQParserPlugin, ChaosMonkeyNothingIsSafeTest, BadComponentTest, BasicDistributedZkTest, SimpleFacetsTest, SuggesterTSTTest, SolrCmdDistributorTest, HdfsDirectoryTest, TestDocSet, TestSolrXml, RecoveryZkTest, TermsComponentTest, TestLMDirichletSimilarityFactory, TestFieldTypeCollectionResource, DeleteInactiveReplicaTest, TestFuzzyAnalyzedSuggestions, TestUtils, TestCoreDiscovery, TestRangeQuery, AnalyticsMergeStrategyTest, FieldFacetTest, CursorMarkTest, TestManagedSynonymFilterFactory, SpellCheckCollatorTest, TestManagedStopFilterFactory, QueryResultKeyTest, MoreLikeThisHandlerTest, XmlUpdateRequestHandlerTest, ShowFileRequestHandlerTest, SpellCheckComponentTest, TestSchemaSimilarityResource, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J0 in 186.83s, 1 test, 1 failure <<< FAILURES!
[...truncated 655 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 417 suites, 1708 tests, 1 failure, 38 ignored (17 assumptions)
Total time: 68 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.7.0_60 -server -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_60) - Build #
10762 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10762/
Java: 64bit/jdk1.7.0_60 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
REGRESSION: org.apache.solr.cloud.OverseerTest.testOverseerFailure
Error Message:
Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
Stack Trace:
org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
at __randomizedtesting.SeedInfo.seed([B9E2DD92FCD61EDC:BDEA5261EE73F1FD]:0)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:143)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:135)
at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:130)
... 47 more
Build Log:
[...truncated 11374 lines...]
[junit4] Suite: org.apache.solr.cloud.OverseerTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-B9E2DD92FCD61EDC-001/init-core-data-001
[junit4] 2> 939317 T3305 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
[junit4] 2> 939317 T3305 oas.SolrTestCaseJ4.initCore ####initCore
[junit4] 2> 939317 T3305 oas.SolrTestCaseJ4.initCore ####initCore end
[junit4] 2> 939320 T3305 oas.SolrTestCaseJ4.setUp ###Starting testDoubleAssignment
[junit4] 2> 939320 T3305 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 939321 T3306 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 939421 T3305 oasc.ZkTestServer.run start zk server on port:40069
[junit4] 2> 939422 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 939424 T3312 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ff62203 name:ZooKeeperConnection Watcher:127.0.0.1:40069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 939425 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 939425 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 939426 T3314 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4eaac9a7 name:ZooKeeperConnection Watcher:127.0.0.1:40069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 939427 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 939430 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 939431 T3316 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e08d82b name:ZooKeeperConnection Watcher:127.0.0.1:40069 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 939432 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 939432 T3305 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 939437 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 939440 T3305 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 939442 T3305 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 939444 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 939448 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 939449 T3318 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58a61f33 name:ZooKeeperConnection Watcher:127.0.0.1:40069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 939450 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 939452 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 939454 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 939458 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 939460 T3320 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@78905076 name:ZooKeeperConnection Watcher:127.0.0.1:40069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 939460 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 939461 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 939464 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 939470 T3305 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:40069_solr
[junit4] 2> 939470 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 939472 T3305 oasc.Overseer.start Overseer (id=92069109050900484-127.0.0.1:40069_solr-n_0000000000) starting
[junit4] 2> 939474 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 939477 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 939480 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 939483 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 939485 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 939488 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 939493 T3322 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 939495 T3321 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 939497 T3321 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 939498 T3321 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 939498 T3321 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 939498 T3321 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 939500 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 939501 T3318 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 939501 T3312 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 939996 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 940001 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 940003 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 940003 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 940004 T3321 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 940005 T3307 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x147185debc40003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 940005 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 940006 T3324 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64e964c1 name:ZooKeeperConnection Watcher:127.0.0.1:40069/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 940006 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 940007 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 940007 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 940008 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 940010 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 940010 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 940011 T3321 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"recovering",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"core_node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 940011 T3321 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 940012 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 940013 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 940015 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 940017 T3320 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 940118 T3312 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 940118 T3324 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 940119 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 940121 T3305 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40069 40069
[junit4] 2> 940661 T3306 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:40069 40069
[junit4] 2> 940662 T3305 oas.SolrTestCaseJ4.tearDown ###Ending testDoubleAssignment
[junit4] 2> 940663 T3321 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:40069_solr
[junit4] 2> 940664 T3325 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 940667 T3305 oas.SolrTestCaseJ4.setUp ###Starting testShardAssignment
[junit4] 2> 940667 T3305 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 940668 T3326 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 940768 T3305 oasc.ZkTestServer.run start zk server on port:36765
[junit4] 2> 940769 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 940771 T3332 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@32472cc8 name:ZooKeeperConnection Watcher:127.0.0.1:36765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 940771 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 940773 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 940774 T3334 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e820d75 name:ZooKeeperConnection Watcher:127.0.0.1:36765 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 940774 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 940775 T3305 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 940777 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 940778 T3336 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d83bcad name:ZooKeeperConnection Watcher:127.0.0.1:36765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 940778 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 940779 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 940780 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 940781 T3338 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50cec160 name:ZooKeeperConnection Watcher:127.0.0.1:36765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 940781 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 940782 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 940783 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 940786 T3305 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36765_solr
[junit4] 2> 940786 T3305 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 940787 T3305 oasc.Overseer.start Overseer (id=92069109139243011-127.0.0.1:36765_solr-n_0000000000) starting
[junit4] 2> 940788 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 940790 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 940792 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 940793 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 940795 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 940797 T3305 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 940799 T3340 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 940800 T3305 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 940800 T3339 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 940801 T3305 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 940802 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 940804 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 940805 T3342 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@799108df name:ZooKeeperConnection Watcher:127.0.0.1:36765/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 940805 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 940806 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 940807 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1
[junit4] 2> 940809 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 940810 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 940810 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 940810 T3339 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 940811 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 940812 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 940812 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 940812 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 941309 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 941317 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 941320 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941321 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 941321 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941323 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core2",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 941323 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 941323 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 941324 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941426 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 941426 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 941821 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 941824 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 941827 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941827 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941827 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941827 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941828 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 941829 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core3",
[junit4] 2> "core_node_name":"node3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 941830 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 941830 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 941831 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 941933 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 941933 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 942328 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 942333 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 942335 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942335 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942336 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942336 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942336 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 942338 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node4",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 942338 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 942338 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 942340 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942441 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 942441 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 942836 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 942842 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942842 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942842 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942843 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 942843 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core5",
[junit4] 2> "core_node_name":"node5",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 942844 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 942844 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 942845 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 942947 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 942947 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943342 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 943349 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 943349 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 943349 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 943349 T3339 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 943350 T3339 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "node_name":"127.0.0.1",
[junit4] 2> "core":"core6",
[junit4] 2> "core_node_name":"node6",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "base_url":"http://127.0.0.1/solr/"}
[junit4] 2> 943350 T3339 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 943350 T3339 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 943351 T3338 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 943453 T3342 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943453 T3336 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 943849 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 943860 T3305 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36765 36765
[junit4] 2> 944414 T3326 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36765 36765
[junit4] 2> 944416 T3305 oas.SolrTestCaseJ4.tearDown ###Ending testShardAssignment
[junit4] 2> 944416 T3339 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:36765_solr
[junit4] 2> 944418 T3343 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 944424 T3305 oas.SolrTestCaseJ4.setUp ###Starting testShardLeaderChange
[junit4] 2> 944425 T3305 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 944426 T3344 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 944526 T3305 oasc.ZkTestServer.run start zk server on port:44512
[junit4] 2> 944527 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 944529 T3350 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f51554e name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 944530 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 944530 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 944532 T3352 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fc4a241 name:ZooKeeperConnection Watcher:127.0.0.1:44512 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 944532 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 944534 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 944535 T3354 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41c9d817 name:ZooKeeperConnection Watcher:127.0.0.1:44512 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 944535 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 944536 T3305 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 944537 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 944540 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 944540 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 944541 T3358 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@556ad759 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 944541 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 944541 T3359 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@59603fde name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 944542 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 944543 T3305 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 944543 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 944545 T3305 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 944545 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 944547 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 944549 T3355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44512_solr
[junit4] 2> 944549 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 944549 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 944550 T3355 oasc.Overseer.start Overseer (id=92069109385527300-127.0.0.1:44512_solr-n_0000000000) starting
[junit4] 2> 944552 T3355 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 944552 T3305 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 944555 T3355 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 944559 T3355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 944563 T3355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 944566 T3355 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 944570 T3355 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 944575 T3355 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4] 2> 944576 T3361 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 944577 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 944579 T3363 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38767d5 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 944579 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 944580 T3361 oasc.OverseerCollectionProcessor.amILeader According to ZK I (id=92069109385527300-127.0.0.1:44512_solr-n_0000000000) am no longer a leader.
[junit4] 2> 944581 T3361 oasc.SolrException.log ERROR :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-queue-work
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:99)
[junit4] 2> at org.apache.solr.cloud.DistributedQueue.getTailId(DistributedQueue.java:483)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:241)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 944581 T3361 oasc.OverseerCollectionProcessor.run ERROR Unable to prioritize overseer org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /roles.json
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:226)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:223)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.prioritizeOverseerNodes(OverseerCollectionProcessor.java:412)
[junit4] 2> at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:256)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 944584 T3355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44512_solr
[junit4] 2> 944584 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 944587 T3355 oasc.Overseer.start Overseer (id=92069109385527301-127.0.0.1:44512_solr-n_0000000001) starting
[junit4] 2> 944600 T3365 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 944601 T3364 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 944602 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 944603 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 944603 T3364 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1]
[junit4] 2> 944603 T3364 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 944605 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 944606 T3358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 945055 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 945060 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 945062 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945063 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 945063 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945063 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 945065 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 945065 T3364 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 945067 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945067 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 945068 T3367 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4326cfea name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 945068 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 945069 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 945070 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 945073 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945073 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 945073 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 945073 T3364 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 945075 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945076 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 945176 T3358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 945177 T3367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 945277 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 945279 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945280 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945280 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945281 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 945281 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 945281 T3364 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4] 2> 945282 T3364 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 945283 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945359 T3342 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard1/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 945359 T3342 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard3/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 945360 T3342 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/collection1/leader_elect/shard2/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 945385 T3358 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 945385 T3367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 945780 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 945785 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 945785 T3367 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 945787 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945787 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945788 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945788 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 945789 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945900 T3367 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 945986 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 945988 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 945990 T3369 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57a8f785 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 945990 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 945991 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 945993 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 945996 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945996 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 945996 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945997 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 945998 T3364 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 945998 T3364 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 945998 T3364 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 945999 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946001 T3369 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 946001 T3355 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4] 2> 946002 T3363 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946002 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946003 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 946004 T3371 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ccba4f1 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 946004 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 946004 T3364 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44512_solr
[junit4] 2> 946007 T3372 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 946009 T3355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44512_solr
[junit4] 2> 946009 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 946009 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 946011 T3374 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f921be7 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 946011 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 946011 T3355 oasc.Overseer.start Overseer (id=92069109385527304-127.0.0.1:44512_solr-n_0000000002) starting
[junit4] 2> 946012 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 946013 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 946015 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946017 T3376 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 946018 T3375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946018 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946019 T3375 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 946019 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946019 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946020 T3374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946020 T3369 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946020 T3375 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 946021 T3375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946022 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946022 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946023 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946025 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946025 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946025 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946027 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946129 T3369 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946129 T3374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946220 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946222 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946222 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946222 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946222 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946223 T3375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946224 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 946224 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946225 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946227 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946227 T3374 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 946232 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946234 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946337 T3374 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 946429 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946431 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 946433 T3378 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a677b49 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 946433 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 946435 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 946439 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 946444 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946444 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946445 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946445 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946447 T3375 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946448 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946448 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946451 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946454 T3378 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 946454 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946454 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946455 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946455 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946456 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946458 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946458 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 946459 T3380 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45eb95a2 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 946459 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 946460 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 946461 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 946464 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946464 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946464 T3375 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946465 T3375 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946466 T3371 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946467 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946517 T3355 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4] 2> 946519 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 946520 T3382 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2eb03948 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 946521 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 946521 T3375 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44512_solr
[junit4] 2> 946522 T3383 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> 946525 T3355 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:44512_solr
[junit4] 2> 946526 T3355 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 946528 T3355 oasc.Overseer.start Overseer (id=92069109385527308-127.0.0.1:44512_solr-n_0000000003) starting
[junit4] 2> 946536 T3385 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 946538 T3384 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946538 T3384 oasc.Overseer$ClusterStateUpdater.run Replaying operations from work queue.
[junit4] 2> 946539 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946539 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946540 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946540 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946543 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946544 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946544 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946544 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946548 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946548 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946551 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946551 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946552 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946552 T3378 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 946554 T3384 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 946668 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946670 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946670 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946672 T3384 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946673 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node2",
[junit4] 2> "core":"core4",
[junit4] 2> "core_node_name":"node2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node2/solr/"}
[junit4] 2> 946673 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946675 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946676 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946676 T3380 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 946678 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946680 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946782 T3380 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 946877 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946880 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 946881 T3387 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ca7fce name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 946882 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 946883 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 946887 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node1
[junit4] 2> 946892 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946892 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946892 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946893 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946894 T3384 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946894 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946895 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946896 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946898 T3387 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 946899 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946899 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946899 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state2",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946899 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946901 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946902 T3305 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 946902 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946903 T3389 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b53f3c1 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 946903 T3305 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 946904 T3305 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 946905 T3305 oascc.SolrZkClient.makePath makePath: /live_nodes/node2
[junit4] 2> 946908 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946908 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 946909 T3384 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"state1",
[junit4] 2> "node_name":"node1",
[junit4] 2> "core":"core1",
[junit4] 2> "core_node_name":"node1",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"1",
[junit4] 2> "base_url":"http://node1/solr/"}
[junit4] 2> 946909 T3384 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4] 2> 946912 T3382 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 946915 T3305 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 946936 T3355 oasc.OverseerTest$OverseerRestarter.run Killing overseer.
[junit4] 2> 946939 T3355 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 946941 T3391 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33e81c41 name:ZooKeeperConnection Watcher:127.0.0.1:44512/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 946941 T3355 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 946942 T3384 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:44512_solr
[junit4] 2> 946944 T3392 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apa
[...truncated too long message...]
ctions/collection1/leaders/shard1
[junit4] 2> 971008 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 972011 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 973013 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 974015 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 975018 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 976020 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 977022 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 978024 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 979027 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 980030 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 981032 T3305 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 981036 T3305 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52309 52309
[junit4] 2> 981828 T3852 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:52309 52309
[junit4] 2> 981829 T3305 oas.SolrTestCaseJ4.tearDown ###Ending testOverseerFailure
[junit4] 2> 981830 T3872 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:52309_solr
[junit4] 2> 981830 T3874 oasc.Overseer$ClusterStateUpdater.checkIfIamStillLeader ERROR could not read the data org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/leader
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:277)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:274)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:317)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater.access$300(Overseer.java:85)
[junit4] 2> at org.apache.solr.cloud.Overseer$ClusterStateUpdater$1.run(Overseer.java:289)
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverseerTest -Dtests.method=testOverseerFailure -Dtests.seed=B9E2DD92FCD61EDC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=el -Dtests.timezone=GB -Dtests.file.encoding=UTF-8
[junit4] ERROR 18.7s J0 | OverseerTest.testOverseerFailure <<<
[junit4] > Throwable #1: org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
[junit4] > at __randomizedtesting.SeedInfo.seed([B9E2DD92FCD61EDC:BDEA5261EE73F1FD]:0)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:143)
[junit4] > at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] > at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:155)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:314)
[junit4] > at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:221)
[junit4] > at org.apache.solr.cloud.OverseerTest$MockZKController.publishState(OverseerTest.java:155)
[junit4] > at org.apache.solr.cloud.OverseerTest.testOverseerFailure(OverseerTest.java:660)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] > Caused by: org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /collections/collection1/leaders/shard1
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:119)
[junit4] > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432)
[junit4] > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:135)
[junit4] > at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
[junit4] > at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:130)
[junit4] > ... 47 more
[junit4] 2> 984839 T3305 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.OverseerTest-B9E2DD92FCD61EDC-001
[junit4] 2> NOTE: test params are: codec=Lucene49: {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=el, timezone=GB
[junit4] 2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.7.0_60 (64-bit)/cpus=8,threads=1,free=107387592,total=390070272
[junit4] 2> NOTE: All tests run in this JVM: [TestStressReorder, BlockCacheTest, ShardRoutingTest, DistributedSpellCheckComponentTest, AnalyticsMergeStrategyTest, TestCoreDiscovery, HdfsWriteToMultipleCollectionsTest, OverseerCollectionProcessorTest, TestFastWriter, ExternalFileFieldSortTest, TestDFRSimilarityFactory, MBeansHandlerTest, HighlighterTest, TestManagedSchema, TestFieldTypeResource, TestManagedResource, SpellingQueryConverterTest, SpellCheckComponentTest, ResponseHeaderTest, TimeZoneUtilsTest, AnalysisErrorHandlingTest, TestFastOutputStream, ReplicationFactorTest, TestFuzzyAnalyzedSuggestions, TestManagedStopFilterFactory, NoCacheHeaderTest, FastVectorHighlighterTest, SimpleFacetsTest, TestStandardQParsers, TestStressVersions, TestSearchPerf, TestPartialUpdateDeduplication, BadComponentTest, XsltUpdateRequestHandlerTest, TestAnalyzedSuggestions, TestDocBasedVersionConstraints, SuggesterWFSTTest, TestChildDocTransformer, TestSort, TestInfoStreamLogging, HttpPartitionTest, DateRangeFieldTest, RAMDirectoryFactoryTest, PathHierarchyTokenizerFactoryTest, CurrencyFieldOpenExchangeTest, ChaosMonkeyNothingIsSafeTest, AliasIntegrationTest, TestJmxIntegration, OverseerStatusTest, TestSchemaSimilarityResource, TestModifyConfFiles, ShowFileRequestHandlerTest, ResponseLogComponentTest, SpellPossibilityIteratorTest, TestCloudManagedSchema, TestCodecSupport, ZkNodePropsTest, TestDistributedSearch, TestComplexPhraseQParserPlugin, NotRequiredUniqueKeyTest, BadCopyFieldTest, BasicFunctionalityTest, TestSolrIndexConfig, HdfsSyncSliceTest, DocumentAnalysisRequestHandlerTest, TestOrdValues, AbstractAnalyticsStatsTest, TestSolr4Spatial, DisMaxRequestHandlerTest, JsonLoaderTest, DistribDocExpirationUpdateProcessorTest, HighlighterMaxOffsetTest, ZkControllerTest, IndexSchemaTest, DocValuesMultiTest, ParsingFieldUpdateProcessorsTest, TestLMDirichletSimilarityFactory, TestLeaderElectionZkExpiry, TestRemoteStreaming, BasicDistributedZkTest, RequiredFieldsTest, SuggesterTest, CoreAdminHandlerTest, FileUtilsTest, ExpressionTest, TestSolrDeletionPolicy1, TestReplicationHandlerBackup, HdfsBasicDistributedZkTest, TestReRankQParserPlugin, PrimitiveFieldTypeTest, TestSchemaResource, TestSolrXml, TestCustomSort, RecoveryZkTest, DateFieldTest, TestDynamicFieldResource, TestSolrJ, FunctionTest, WordBreakSolrSpellCheckerTest, StatelessScriptUpdateProcessorFactoryTest, SyncSliceTest, OverseerTest]
[junit4] Completed on J0 in 45.55s, 9 tests, 1 error <<< FAILURES!
[...truncated 619 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 417 suites, 1708 tests, 1 error, 38 ignored (17 assumptions)
Total time: 67 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.7.0_60 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_05) - Build #
10761 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10761/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseParallelGC
2 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard2 is not consistent. Got 220 from https://127.0.0.1:57716/rm_/o/collection1lastClient and got 165 from https://127.0.0.1:39035/rm_/o/collection1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 220 from https://127.0.0.1:57716/rm_/o/collection1lastClient and got 165 from https://127.0.0.1:39035/rm_/o/collection1
at __randomizedtesting.SeedInfo.seed([AAFB06BE7262B15:8B493E7390794B29]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1132)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1111)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.solr.cloud.RollingRestartTest.testDistribSearch
Error Message:
Unable to restart (#1): CloudJettyRunner [url=https://127.0.0.1:54317/_vn/gc/collection1]
Stack Trace:
java.lang.AssertionError: Unable to restart (#1): CloudJettyRunner [url=https://127.0.0.1:54317/_vn/gc/collection1]
at __randomizedtesting.SeedInfo.seed([AAFB06BE7262B15:8B493E7390794B29]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:102)
at org.apache.solr.cloud.RollingRestartTest.doTest(RollingRestartTest.java:61)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 11271 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/init-core-data-001
[junit4] 2> 682739 T3421 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
[junit4] 2> 682740 T3421 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /rm_/o
[junit4] 2> 682744 T3421 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 682745 T3421 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 682745 T3422 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 682845 T3421 oasc.ZkTestServer.run start zk server on port:52362
[junit4] 2> 682846 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 682848 T3428 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b0954aa name:ZooKeeperConnection Watcher:127.0.0.1:52362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 682848 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 682849 T3421 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 682851 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 682852 T3430 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1eeccf84 name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 682852 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 682852 T3421 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 682854 T3421 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 682855 T3421 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 682855 T3421 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 682856 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 682857 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 682858 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 682858 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 682859 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 682859 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 682860 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 682861 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 682862 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 682862 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 682863 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 682863 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 682864 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 682864 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 682865 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 682866 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 682866 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 682867 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 682868 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 682868 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 682869 T3421 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 682870 T3421 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 682871 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 682872 T3432 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12cb7afa name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 682872 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 683092 T3421 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 683094 T3421 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 683097 T3421 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:58033
[junit4] 2> 683098 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 683098 T3421 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 683099 T3421 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002
[junit4] 2> 683099 T3421 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/'
[junit4] 2> 683110 T3421 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/solr.xml
[junit4] 2> 683127 T3421 oasc.CoreContainer.<init> New CoreContainer 1285219428
[junit4] 2> 683127 T3421 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/]
[junit4] 2> 683128 T3421 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 683128 T3421 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 683128 T3421 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 683129 T3421 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 683130 T3421 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 683131 T3421 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 683131 T3421 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 683132 T3421 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 683132 T3421 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52362/solr
[junit4] 2> 683132 T3421 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 683133 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 683134 T3443 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1739d36d name:ZooKeeperConnection Watcher:127.0.0.1:52362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 683134 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 683136 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 683137 T3445 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c03e184 name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 683137 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 683138 T3421 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 683139 T3421 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 683141 T3421 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 683142 T3421 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 683144 T3421 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 683145 T3421 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 683146 T3421 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58033_rm_%2Fo
[junit4] 2> 683147 T3421 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58033_rm_%2Fo
[junit4] 2> 683148 T3421 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 683149 T3421 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 683151 T3421 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:58033_rm_%2Fo
[junit4] 2> 683151 T3421 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 683152 T3421 oasc.Overseer.start Overseer (id=92068493388873732-127.0.0.1:58033_rm_%2Fo-n_0000000000) starting
[junit4] 2> 683153 T3421 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 683158 T3447 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 683158 T3421 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 683159 T3421 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 683160 T3421 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 683161 T3446 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 683163 T3448 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 683163 T3448 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 683163 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 683163 T3448 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 683164 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 683164 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:58033/rm_/o",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58033_rm_%2Fo",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 683165 T3446 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 683165 T3446 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 683166 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 683166 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 684164 T3448 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 684164 T3448 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 684165 T3448 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 684165 T3448 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 684165 T3448 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 684165 T3448 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/'
[junit4] 2> 684166 T3448 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 684166 T3448 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 684166 T3448 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 684182 T3448 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 684191 T3448 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 684192 T3448 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 684196 T3448 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 684288 T3448 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 684289 T3448 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 684290 T3448 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 684292 T3448 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 684302 T3448 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 684303 T3448 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 684312 T3448 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/
[junit4] 2> 684312 T3448 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2dc8a1f9
[junit4] 2> 684313 T3448 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data
[junit4] 2> 684313 T3448 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index/
[junit4] 2> 684313 T3448 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 684313 T3448 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index
[junit4] 2> 684314 T3448 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=59.9013671875, floorSegmentMB=1.90234375, forceMergeDeletesPctAllowed=19.856637667348704, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 693751 T3448 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 693751 T3448 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 693754 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 693754 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 693754 T3448 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 693754 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 693754 T3448 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 693754 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 693755 T3448 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 693755 T3448 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 693755 T3448 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 693755 T3448 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 693755 T3448 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 693756 T3448 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 693757 T3448 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 693758 T3448 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 693759 T3448 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 693759 T3448 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 693759 T3448 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 693760 T3448 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 693760 T3448 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 693760 T3448 oass.SolrIndexSearcher.<init> Opening Searcher@3cb8561e[collection1] main
[junit4] 2> 693761 T3448 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 693761 T3448 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 693761 T3448 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 693762 T3448 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 693762 T3448 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 693762 T3448 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 693762 T3448 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 693763 T3448 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 693764 T3448 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 693765 T3448 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 693765 T3448 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 693765 T3448 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 693768 T3449 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3cb8561e[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 693768 T3448 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 693769 T3452 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:58033/rm_/o collection:control_collection shard:shard1
[junit4] 2> 693769 T3421 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 693769 T3452 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 693769 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 693772 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 693772 T3452 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 693772 T3454 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56e26ba4 name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 693773 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 693773 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693773 T3452 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 693773 T3452 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 693773 T3421 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 693773 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C908 name=collection1 org.apache.solr.core.SolrCore@29733922 url=https://127.0.0.1:58033/rm_/o/collection1 node=127.0.0.1:58033_rm_%2Fo C908_STATE=coll:control_collection core:collection1 props:{state=down, base_url=https://127.0.0.1:58033/rm_/o, core=collection1, node_name=127.0.0.1:58033_rm_%2Fo}
[junit4] 2> 693773 T3452 C908 P58033 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:58033/rm_/o/collection1/
[junit4] 2> 693774 T3452 C908 P58033 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 693774 T3452 C908 P58033 oasc.SyncStrategy.syncToMe https://127.0.0.1:58033/rm_/o/collection1/ has no replicas
[junit4] 2> 693774 T3452 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:58033/rm_/o/collection1/ shard1
[junit4] 2> 693774 T3421 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 693775 T3452 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 693775 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693775 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693775 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693777 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693778 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 693779 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693880 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693880 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 693928 T3452 oasc.ZkController.register We are https://127.0.0.1:58033/rm_/o/collection1/ and leader is https://127.0.0.1:58033/rm_/o/collection1/
[junit4] 2> 693928 T3452 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:58033/rm_/o
[junit4] 2> 693929 T3452 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 693929 T3452 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 693929 T3452 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 693930 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693930 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693930 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 693930 T3452 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 693931 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 693931 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:58033/rm_/o",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:58033_rm_%2Fo",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 693934 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 694005 T3421 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 694006 T3421 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 694008 T3421 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 694009 T3421 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:60624
[junit4] 2> 694011 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 694011 T3421 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 694011 T3421 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003
[junit4] 2> 694011 T3421 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/'
[junit4] 2> 694021 T3421 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/solr.xml
[junit4] 2> 694028 T3421 oasc.CoreContainer.<init> New CoreContainer 350194343
[junit4] 2> 694028 T3421 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/]
[junit4] 2> 694029 T3421 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 694029 T3421 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 694029 T3421 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 694029 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 694030 T3421 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 694032 T3421 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 694032 T3421 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 694033 T3421 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 694033 T3421 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52362/solr
[junit4] 2> 694033 T3421 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 694034 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 694035 T3465 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e6c2ae1 name:ZooKeeperConnection Watcher:127.0.0.1:52362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 694035 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 694035 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 694035 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 694036 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 694037 T3467 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d29f73e name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 694037 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 694040 T3421 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 695042 T3421 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60624_rm_%2Fo
[junit4] 2> 695043 T3421 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60624_rm_%2Fo
[junit4] 2> 695052 T3468 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 695052 T3468 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 695052 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 695052 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 695053 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 695052 T3468 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 695053 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 695054 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:60624/rm_/o",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60624_rm_%2Fo",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 695054 T3446 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 695054 T3446 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 695055 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 695157 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 695157 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 695157 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696053 T3468 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 696053 T3468 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 696054 T3468 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 696054 T3468 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 696055 T3468 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 696055 T3468 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/'
[junit4] 2> 696056 T3468 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 696056 T3468 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 696056 T3468 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 696081 T3468 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 696090 T3468 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 696091 T3468 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 696095 T3468 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 696173 T3468 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 696174 T3468 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 696175 T3468 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 696177 T3468 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 696187 T3468 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 696187 T3468 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 696187 T3468 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/
[junit4] 2> 696187 T3468 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2dc8a1f9
[junit4] 2> 696188 T3468 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1
[junit4] 2> 696188 T3468 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index/
[junit4] 2> 696189 T3468 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 696189 T3468 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index
[junit4] 2> 696189 T3468 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=59.9013671875, floorSegmentMB=1.90234375, forceMergeDeletesPctAllowed=19.856637667348704, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 696198 T3468 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 696198 T3468 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 696201 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 696201 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 696201 T3468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 696202 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 696202 T3468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 696202 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 696202 T3468 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 696202 T3468 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 696202 T3468 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 696203 T3468 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 696204 T3468 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 696204 T3468 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 696204 T3468 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 696206 T3468 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 696207 T3468 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 696207 T3468 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 696207 T3468 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 696208 T3468 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 696208 T3468 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 696208 T3468 oass.SolrIndexSearcher.<init> Opening Searcher@62920d54[collection1] main
[junit4] 2> 696208 T3468 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 696209 T3468 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 696210 T3468 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 696210 T3468 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 696210 T3468 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 696210 T3468 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 696211 T3468 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 696211 T3468 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 696211 T3468 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 696211 T3468 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 696213 T3469 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62920d54[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 696214 T3468 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 696215 T3472 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:60624/rm_/o collection:collection1 shard:shard2
[junit4] 2> 696215 T3421 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 696215 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 696215 T3472 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 696218 T3472 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 696219 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696219 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696219 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696220 T3472 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 696220 T3472 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 696220 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C909 name=collection1 org.apache.solr.core.SolrCore@5e375d0e url=https://127.0.0.1:60624/rm_/o/collection1 node=127.0.0.1:60624_rm_%2Fo C909_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:60624/rm_/o, core=collection1, node_name=127.0.0.1:60624_rm_%2Fo}
[junit4] 2> 696220 T3472 C909 P60624 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:60624/rm_/o/collection1/
[junit4] 2> 696220 T3472 C909 P60624 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 696221 T3472 C909 P60624 oasc.SyncStrategy.syncToMe https://127.0.0.1:60624/rm_/o/collection1/ has no replicas
[junit4] 2> 696221 T3472 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:60624/rm_/o/collection1/ shard2
[junit4] 2> 696221 T3472 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 696221 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696225 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696226 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696327 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696327 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696327 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696376 T3472 oasc.ZkController.register We are https://127.0.0.1:60624/rm_/o/collection1/ and leader is https://127.0.0.1:60624/rm_/o/collection1/
[junit4] 2> 696376 T3472 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:60624/rm_/o
[junit4] 2> 696376 T3472 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 696376 T3472 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 696376 T3472 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 696378 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696378 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696378 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696378 T3472 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 696379 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 696379 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:60624/rm_/o",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60624_rm_%2Fo",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 696380 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 696437 T3421 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 696438 T3421 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 696440 T3421 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 696441 T3421 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:42159
[junit4] 2> 696443 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 696443 T3421 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 696443 T3421 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004
[junit4] 2> 696443 T3421 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/'
[junit4] 2> 696453 T3421 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/solr.xml
[junit4] 2> 696460 T3421 oasc.CoreContainer.<init> New CoreContainer 1136505569
[junit4] 2> 696461 T3421 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/]
[junit4] 2> 696461 T3421 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 696461 T3421 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 696461 T3421 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 696462 T3421 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 696463 T3421 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 696464 T3421 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 696464 T3421 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 696465 T3421 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 696465 T3421 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:52362/solr
[junit4] 2> 696465 T3421 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 696466 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 696467 T3483 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f29bc8c name:ZooKeeperConnection Watcher:127.0.0.1:52362 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 696467 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 696469 T3421 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 696469 T3485 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e1acc85 name:ZooKeeperConnection Watcher:127.0.0.1:52362/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 696469 T3421 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 696473 T3421 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 696481 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696481 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696481 T3485 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 696481 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 697474 T3421 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42159_rm_%2Fo
[junit4] 2> 697475 T3421 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42159_rm_%2Fo
[junit4] 2> 697479 T3486 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 697480 T3486 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 697481 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 697481 T3486 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 697481 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 697481 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 697481 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 697482 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:42159/rm_/o",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42159_rm_%2Fo",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 697482 T3446 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 697482 T3446 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 697483 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 697584 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 697584 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 697584 T3485 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 697584 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 698481 T3486 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 698482 T3486 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 698482 T3486 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 698482 T3486 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 698483 T3486 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 698483 T3486 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/'
[junit4] 2> 698484 T3486 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 698484 T3486 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 698485 T3486 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 698507 T3486 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 698524 T3486 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 698525 T3486 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 698531 T3486 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 698613 T3486 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 698615 T3486 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 698615 T3486 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 698617 T3486 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 698629 T3486 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 698629 T3486 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 698629 T3486 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/
[junit4] 2> 698629 T3486 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2dc8a1f9
[junit4] 2> 698630 T3486 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2
[junit4] 2> 698630 T3486 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index/
[junit4] 2> 698630 T3486 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 698630 T3486 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index
[junit4] 2> 698631 T3486 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=16, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=59.9013671875, floorSegmentMB=1.90234375, forceMergeDeletesPctAllowed=19.856637667348704, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 698645 T3486 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 698645 T3486 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 698648 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 698648 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 698648 T3486 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 698648 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 698648 T3486 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 698649 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 698649 T3486 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 698649 T3486 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 698649 T3486 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 698649 T3486 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 698650 T3486 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 698651 T3486 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 698651 T3486 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 698652 T3486 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 698653 T3486 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 698653 T3486 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 698654 T3486 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=36, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 698655 T3486 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 698655 T3486 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 698655 T3486 oass.SolrIndexSearcher.<init> Opening Searcher@6d7f5127[collection1] main
[junit4] 2> 698655 T3486 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 698656 T3486 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 698656 T3486 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 698657 T3486 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 698657 T3486 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 698657 T3486 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 698658 T3486 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 698658 T3486 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 698658 T3486 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 698658 T3486 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 698660 T3487 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6d7f5127[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 698661 T3486 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 698662 T3490 oasc.ZkController.register Register replica - core:collection1 address:https://127.0.0.1:42159/rm_/o collection:collection1 shard:shard3
[junit4] 2> 698662 T3421 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 698662 T3490 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 698662 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 698665 T3490 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 698665 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698666 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698666 T3490 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 698666 T3490 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 698666 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 698666 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C910 name=collection1 org.apache.solr.core.SolrCore@51c3a3c9 url=https://127.0.0.1:42159/rm_/o/collection1 node=127.0.0.1:42159_rm_%2Fo C910_STATE=coll:collection1 core:collection1 props:{state=down, base_url=https://127.0.0.1:42159/rm_/o, core=collection1, node_name=127.0.0.1:42159_rm_%2Fo}
[junit4] 2> 698666 T3490 C910 P42159 oasc.SyncStrategy.sync Sync replicas to https://127.0.0.1:42159/rm_/o/collection1/
[junit4] 2> 698667 T3490 C910 P42159 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 698667 T3490 C910 P42159 oasc.SyncStrategy.syncToMe https://127.0.0.1:42159/rm_/o/collection1/ has no replicas
[junit4] 2> 698667 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698667 T3490 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: https://127.0.0.1:42159/rm_/o/collection1/ shard3
[junit4] 2> 698668 T3490 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 698669 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698671 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698772 T3454 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 698772 T3445 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 698772 T3485 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 698772 T3467 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 698820 T3490 oasc.ZkController.register We are https://127.0.0.1:42159/rm_/o/collection1/ and leader is https://127.0.0.1:42159/rm_/o/collection1/
[junit4] 2> 698820 T3490 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=https://127.0.0.1:42159/rm_/o
[junit4] 2> 698821 T3490 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 698821 T3490 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 698821 T3490 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 698822 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698822 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698822 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698822 T3490 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 698823 T3446 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 698824 T3446 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"https://127.0.0.1:42159/rm_/o",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:42159_rm_%2Fo",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 698825 T3445 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 698887 T3421 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 698887 T3421 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 698890 T3421 oejus.SslContextFactory.doStart Enabled Protocols [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]
[junit4] 2> 698890 T3421 oejs.AbstractConnector.doStart Started SslSelectChannelConnector@127.0.0.1:37470
[junit4] 2> 698892 T3421 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 698892 T3421 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 698893 T3421 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-AAFB06BE7262B15-001/tempDir-005
[junit4] 2> 698893 T3421 oas
[...truncated too long message...]
losing IndexWriter
[junit4] 2> 1083433 T3918 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 1083434 T3918 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 1083435 T3918 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 1083435 T3918 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15;done=false>>]
[junit4] 2> 1083435 T3918 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15
[junit4] 2> 1083435 T3918 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15/index;done=false>>]
[junit4] 2> 1083436 T3918 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001/tempDir-001/jetty15/index
[junit4] 2> 1083436 T3918 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 1083437 T3918 oasc.SolrException.log ERROR There was a problem trying to register as the leader:org.apache.solr.common.SolrException: Could not register as the leader because creating the ephemeral registration node in ZooKeeper failed
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:143)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:298)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:163)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:125)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:209)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.execute(SolrZkClient.java:206)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:206)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:417)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:386)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:373)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:135)
[junit4] 2> at org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:31)
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:130)
[junit4] 2> ... 7 more
[junit4] 2>
[junit4] 2> 1083437 T3918 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 1083438 T3918 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./temp/solr.cloud.RollingRestartTest-AAFB06BE7262B15-001
[junit4] 2> NOTE: test params are: codec=Lucene45, sim=DefaultSimilarity, locale=nl_NL, timezone=US/Samoa
[junit4] 2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=142840936,total=317194240
[junit4] 2> NOTE: All tests run in this JVM: [TestDistribDocBasedVersion, DirectSolrSpellCheckerTest, TestFoldingMultitermQuery, TestBlendedInfixSuggestions, TestWordDelimiterFilterFactory, XmlUpdateRequestHandlerTest, TestCodecSupport, BasicDistributedZkTest, SolrCoreCheckLockOnStartupTest, PluginInfoTest, JsonLoaderTest, BasicFunctionalityTest, TestSolr4Spatial, TestTrie, BadIndexSchemaTest, PingRequestHandlerTest, TestComponentsName, TestHighFrequencyDictionaryFactory, TestCollectionAPI, RegexBoostProcessorTest, ReturnFieldsTest, StressHdfsTest, PreAnalyzedUpdateProcessorTest, TestNonNRTOpen, SpellPossibilityIteratorTest, FieldMutatingUpdateProcessorTest, XsltUpdateRequestHandlerTest, SuggesterWFSTTest, SignatureUpdateProcessorFactoryTest, TestFieldTypeCollectionResource, ShowFileRequestHandlerTest, FieldFacetTest, ModifyConfFileTest, SimpleFacetsTest, SolrIndexSplitterTest, TestDynamicFieldCollectionResource, ShardRoutingTest, TestDocBasedVersionConstraints, OpenExchangeRatesOrgProviderTest, TestUniqueKeyFieldResource, SSLMigrationTest, UnloadDistributedZkTest, SuggestComponentTest, TestFastLRUCache, HdfsWriteToMultipleCollectionsTest, RankQueryTest, HdfsRecoveryZkTest, HdfsCollectionsAPIDistributedZkTest, HdfsBasicDistributedZk2Test, ExternalFileFieldSortTest, TestSchemaResource, PathHierarchyTokenizerFactoryTest, HighlighterMaxOffsetTest, JSONWriterTest, DocValuesMissingTest, ChaosMonkeyNothingIsSafeTest, TestStressRecovery, HttpPartitionTest, StandardRequestHandlerTest, SyncSliceTest, TestHighlightDedupGrouping, BadComponentTest, TestConfigSets, IndexSchemaTest, ZkNodePropsTest, TestManagedSynonymFilterFactory, HdfsDirectoryTest, FastVectorHighlighterTest, CustomCollectionTest, ConnectionManagerTest, SpellCheckCollatorTest, TestSolrDeletionPolicy2, MigrateRouteKeyTest, TestFastWriter, DocValuesMultiTest, TestDocSet, ZkCLITest, ZkControllerTest, TestMaxScoreQueryParser, TestShardHandlerFactory, TestManagedSchema, ResponseLogComponentTest, SolrCmdDistributorTest, TestReload, TestAddFieldRealTimeGet, ShardSplitTest, FieldFacetExtrasTest, TestStressReorder, TestCloudInspectUtil, CurrencyFieldOpenExchangeTest, RollingRestartTest]
[junit4] Completed on J0 in 98.71s, 1 test, 1 failure <<< FAILURES!
[...truncated 638 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 417 suites, 1708 tests, 2 failures, 38 ignored (17 assumptions)
Total time: 62 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Re: [JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build #
10760 - Still Failing!
Posted by "david.w.smiley@gmail.com" <da...@gmail.com>.
I’m on it; this’ll get fixed momentarily. Some co-related JIRA issues; one
got committed without the other.
~ David Smiley
Freelance Apache Lucene/Solr Search Consultant/Developer
http://www.linkedin.com/in/davidwsmiley
On Tue, Jul 8, 2014 at 2:54 PM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:
> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10760/
> Java: 32bit/ibm-j9-jdk7
> -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
>
> 2 tests failed.
> FAILED: org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations
> {#5 seed=[CDF4E70BC4C20419:69B5EDAA7D26DA74]}
>
> Error Message:
> [Contains] Shouldn't match
> I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0)
> Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
>
> Stack Trace:
> java.lang.AssertionError: [Contains] Shouldn't match
> I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0)
> Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
> at
> __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:69B5EDAA7D26DA74]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
> at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
> at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
> at
> org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:619)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:853)
>
>
> FAILED: org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations
> {#7 seed=[CDF4E70BC4C20419:CBD813564C025EDC]}
>
> Error Message:
> [Contains] Shouldn't match
> I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0)
> Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
>
> Stack Trace:
> java.lang.AssertionError: [Contains] Shouldn't match
> I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0)
> Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
> at
> __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:CBD813564C025EDC]:0)
> at org.junit.Assert.fail(Assert.java:93)
> at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
> at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
> at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
> at
> org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
> at java.lang.reflect.Method.invoke(Method.java:619)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
> at
> com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
> at
> org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
> at java.lang.Thread.run(Thread.java:853)
>
>
>
>
> Build Log:
> [...truncated 9432 lines...]
> [junit4] Suite: org.apache.lucene.spatial.bbox.TestBBoxStrategy
> [junit4] 2> ??? 09, 2014 2:53:26 ?
> org.apache.lucene.spatial.StrategyTestCase executeQueries
> [junit4] 2> INFO: testing queried for strategy BBoxStrategy
> field:bbox ctx=SpatialContext.GEO
> [junit4] 2> ??? 09, 2014 2:53:26 ?
> org.apache.lucene.spatial.StrategyTestCase executeQueries
> [junit4] 2> INFO: testing queried for strategy BBoxStrategy
> field:bbox ctx=SpatialContext.GEO
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations
> -Dtests.seed=CDF4E70BC4C20419 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-8
> -Dtests.file.encoding=US-ASCII
> [junit4] FAILURE 0.22s J1 | TestBBoxStrategy.testOperations {#5
> seed=[CDF4E70BC4C20419:69B5EDAA7D26DA74]} <<<
> [junit4] > Throwable #1: java.lang.AssertionError: [Contains]
> Shouldn't match I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0)
> Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:69B5EDAA7D26DA74]:0)
> [junit4] > at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
> [junit4] > at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
> [junit4] > at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
> [junit4] > at
> org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
> [junit4] > at java.lang.Thread.run(Thread.java:853)
> [junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations
> -Dtests.seed=CDF4E70BC4C20419 -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-8
> -Dtests.file.encoding=US-ASCII
> [junit4] FAILURE 0.19s J1 | TestBBoxStrategy.testOperations {#7
> seed=[CDF4E70BC4C20419:CBD813564C025EDC]} <<<
> [junit4] > Throwable #1: java.lang.AssertionError: [Contains]
> Shouldn't match I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0)
> Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:CBD813564C025EDC]:0)
> [junit4] > at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
> [junit4] > at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
> [junit4] > at
> org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
> [junit4] > at
> org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
> [junit4] > at java.lang.Thread.run(Thread.java:853)
> [junit4] 2> ??? 09, 2014 2:53:35 ?
> org.apache.lucene.spatial.StrategyTestCase executeQueries
> [junit4] 2> INFO: testing queried for strategy BBoxStrategy
> field:bbox ctx=SpatialContext.GEO
> [junit4] 2> NOTE: leaving temporary files on disk at:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/spatial/test/J1/./temp/lucene.spatial.bbox.TestBBoxStrategy-CDF4E70BC4C20419-001
> [junit4] 2> NOTE: test params are: codec=Lucene40,
> sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {bbox__minY=DFR
> I(n)B3(800.0), bbox__maxX=IB SPL-D1, bbox__maxY=IB LL-D1, bbox__xdl=DFR
> GZ(0.3), bbox__minX=DFR G1}, locale=ar_KW, timezone=Etc/GMT-8
> [junit4] 2> NOTE: Linux 3.8.0-42-generic x86/IBM Corporation 1.7.0
> (32-bit)/cpus=8,threads=1,free=24357064,total=56819712
> [junit4] 2> NOTE: All tests run in this JVM: [JtsPolygonTest,
> TestTermQueryPrefixGridStrategy, TestPointVectorStrategy,
> DistanceStrategyTest, SpatialPrefixTreeTest, TestBBoxStrategy]
> [junit4] Completed on J1 in 10.26s, 30 tests, 2 failures, 1 skipped <<<
> FAILURES!
>
> [...truncated 20 lines...]
> BUILD FAILED
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:460: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The
> following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:544:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2019:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298:
> The following error occurred while executing this line:
> /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922:
> There were test failures: 16 suites, 143 tests, 2 failures, 13 ignored (2
> assumptions)
>
> Total time: 39 minutes 55 seconds
> Build step 'Invoke Ant' marked build as failure
> [description-setter] Description set: Java: 32bit/ibm-j9-jdk7
> -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
[JENKINS] Lucene-Solr-trunk-Linux (32bit/ibm-j9-jdk7) - Build #
10760 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10760/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
2 tests failed.
FAILED: org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#5 seed=[CDF4E70BC4C20419:69B5EDAA7D26DA74]}
Error Message:
[Contains] Shouldn't match I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
Stack Trace:
java.lang.AssertionError: [Contains] Shouldn't match I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
at __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:69B5EDAA7D26DA74]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:853)
FAILED: org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#7 seed=[CDF4E70BC4C20419:CBD813564C025EDC]}
Error Message:
[Contains] Shouldn't match I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0) Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
Stack Trace:
java.lang.AssertionError: [Contains] Shouldn't match I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0) Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
at __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:CBD813564C025EDC]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:94)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:619)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:853)
Build Log:
[...truncated 9432 lines...]
[junit4] Suite: org.apache.lucene.spatial.bbox.TestBBoxStrategy
[junit4] 2> ??? 09, 2014 2:53:26 ? org.apache.lucene.spatial.StrategyTestCase executeQueries
[junit4] 2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
[junit4] 2> ??? 09, 2014 2:53:26 ? org.apache.lucene.spatial.StrategyTestCase executeQueries
[junit4] 2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=CDF4E70BC4C20419 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-8 -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 0.22s J1 | TestBBoxStrategy.testOperations {#5 seed=[CDF4E70BC4C20419:69B5EDAA7D26DA74]} <<<
[junit4] > Throwable #1: java.lang.AssertionError: [Contains] Shouldn't match I#5:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=-110.0,maxX=-100.0,minY=-90.0,maxY=-90.0)
[junit4] > at __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:69B5EDAA7D26DA74]:0)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
[junit4] > at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
[junit4] > at java.lang.Thread.run(Thread.java:853)
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=CDF4E70BC4C20419 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-8 -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 0.19s J1 | TestBBoxStrategy.testOperations {#7 seed=[CDF4E70BC4C20419:CBD813564C025EDC]} <<<
[junit4] > Throwable #1: java.lang.AssertionError: [Contains] Shouldn't match I#0:Rect(minX=-150.0,maxX=-50.0,minY=-70.0,maxY=-70.0) Q:Rect(minX=-150.0,maxX=-130.0,minY=-70.0,maxY=-70.0)
[junit4] > at __randomizedtesting.SeedInfo.seed([CDF4E70BC4C20419:CBD813564C025EDC]:0)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
[junit4] > at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
[junit4] > at java.lang.Thread.run(Thread.java:853)
[junit4] 2> ??? 09, 2014 2:53:35 ? org.apache.lucene.spatial.StrategyTestCase executeQueries
[junit4] 2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/spatial/test/J1/./temp/lucene.spatial.bbox.TestBBoxStrategy-CDF4E70BC4C20419-001
[junit4] 2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {bbox__minY=DFR I(n)B3(800.0), bbox__maxX=IB SPL-D1, bbox__maxY=IB LL-D1, bbox__xdl=DFR GZ(0.3), bbox__minX=DFR G1}, locale=ar_KW, timezone=Etc/GMT-8
[junit4] 2> NOTE: Linux 3.8.0-42-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=24357064,total=56819712
[junit4] 2> NOTE: All tests run in this JVM: [JtsPolygonTest, TestTermQueryPrefixGridStrategy, TestPointVectorStrategy, DistanceStrategyTest, SpatialPrefixTreeTest, TestBBoxStrategy]
[junit4] Completed on J1 in 10.26s, 30 tests, 2 failures, 1 skipped <<< FAILURES!
[...truncated 20 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:460: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:544: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2019: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 16 suites, 143 tests, 2 failures, 13 ignored (2 assumptions)
Total time: 39 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_05) - Build #
10759 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10759/
Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
3 tests failed.
FAILED: org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#4 seed=[D219D8E1E28D9A88:E7A2011F26D509E5]}
Error Message:
[Contains] Shouldn't match I#1:Rect(minX=50.0,maxX=120.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=80.0,maxX=80.0,minY=-90.0,maxY=-90.0)
Stack Trace:
java.lang.AssertionError: [Contains] Shouldn't match I#1:Rect(minX=50.0,maxX=120.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=80.0,maxX=80.0,minY=-90.0,maxY=-90.0)
at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:E7A2011F26D509E5]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#9 seed=[D219D8E1E28D9A88:31129F6879D1E045]}
Error Message:
[IsWithin] Shouldn't match I#0:Rect(minX=270.0,maxX=280.0,minY=-80.0,maxY=-80.0) Q:Rect(minX=170.0,maxX=300.0,minY=-80.0,maxY=-80.0)
Stack Trace:
java.lang.AssertionError: [IsWithin] Shouldn't match I#0:Rect(minX=270.0,maxX=280.0,minY=-80.0,maxY=-80.0) Q:Rect(minX=170.0,maxX=300.0,minY=-80.0,maxY=-80.0)
at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:31129F6879D1E045]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations {#10 seed=[D219D8E1E28D9A88:C4537733AB6DC29D]}
Error Message:
[IsWithin] Shouldn't match I#3:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=40.0,maxX=-60.0,minY=-90.0,maxY=-90.0)
Stack Trace:
java.lang.AssertionError: [IsWithin] Shouldn't match I#3:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=40.0,maxX=-60.0,minY=-90.0,maxY=-90.0)
at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:C4537733AB6DC29D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 9631 lines...]
[junit4] Suite: org.apache.lucene.spatial.bbox.TestBBoxStrategy
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=D219D8E1E28D9A88 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=Asia/Seoul -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.17s J1 | TestBBoxStrategy.testOperations {#4 seed=[D219D8E1E28D9A88:E7A2011F26D509E5]} <<<
[junit4] > Throwable #1: java.lang.AssertionError: [Contains] Shouldn't match I#1:Rect(minX=50.0,maxX=120.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=80.0,maxX=80.0,minY=-90.0,maxY=-90.0)
[junit4] > at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:E7A2011F26D509E5]:0)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
[junit4] > at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=D219D8E1E28D9A88 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=Asia/Seoul -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.11s J1 | TestBBoxStrategy.testOperations {#9 seed=[D219D8E1E28D9A88:31129F6879D1E045]} <<<
[junit4] > Throwable #1: java.lang.AssertionError: [IsWithin] Shouldn't match I#0:Rect(minX=270.0,maxX=280.0,minY=-80.0,maxY=-80.0) Q:Rect(minX=170.0,maxX=300.0,minY=-80.0,maxY=-80.0)
[junit4] > at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:31129F6879D1E045]:0)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
[junit4] > at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBBoxStrategy -Dtests.method=testOperations -Dtests.seed=D219D8E1E28D9A88 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw_IL -Dtests.timezone=Asia/Seoul -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.19s J1 | TestBBoxStrategy.testOperations {#10 seed=[D219D8E1E28D9A88:C4537733AB6DC29D]} <<<
[junit4] > Throwable #1: java.lang.AssertionError: [IsWithin] Shouldn't match I#3:Rect(minX=-150.0,maxX=-80.0,minY=-90.0,maxY=-90.0) Q:Rect(minX=40.0,maxX=-60.0,minY=-90.0,maxY=-90.0)
[junit4] > at __randomizedtesting.SeedInfo.seed([D219D8E1E28D9A88:C4537733AB6DC29D]:0)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.fail(RandomSpatialOpStrategyTestCase.java:126)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperation(RandomSpatialOpStrategyTestCase.java:115)
[junit4] > at org.apache.lucene.spatial.prefix.RandomSpatialOpStrategyTestCase.testOperationRandomShapes(RandomSpatialOpStrategyTestCase.java:62)
[junit4] > at org.apache.lucene.spatial.bbox.TestBBoxStrategy.testOperations(TestBBoxStrategy.java:97)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> יול 09, 2014 2:03:24 AM org.apache.lucene.spatial.StrategyTestCase executeQueries
[junit4] 2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
[junit4] 2> יול 09, 2014 2:03:24 AM org.apache.lucene.spatial.StrategyTestCase executeQueries
[junit4] 2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
[junit4] 2> יול 09, 2014 2:03:24 AM org.apache.lucene.spatial.StrategyTestCase executeQueries
[junit4] 2> INFO: testing queried for strategy BBoxStrategy field:bbox ctx=SpatialContext.GEO
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build/spatial/test/J1/./temp/lucene.spatial.bbox.TestBBoxStrategy-D219D8E1E28D9A88-001
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {bbox__xdl=LM Jelinek-Mercer(0.100000), bbox__maxX=DFR I(n)2, bbox__maxY=DFR GB3(800.0), bbox__minX=DFR GL3(800.0), bbox__minY=DFR GL1}, locale=iw_IL, timezone=Asia/Seoul
[junit4] 2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=133908160,total=189792256
[junit4] 2> NOTE: All tests run in this JVM: [SpatialArgsParserTest, SpatialPrefixTreeTest, TestPointVectorStrategy, DateNRStrategyTest, TestBBoxStrategy]
[junit4] Completed on J1 in 4.90s, 30 tests, 3 failures, 1 skipped <<< FAILURES!
[...truncated 12 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/build.xml:544: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:2019: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/module-build.xml:60: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 16 suites, 143 tests, 3 failures, 13 ignored (2 assumptions)
Total time: 40 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_05 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_20-ea-b21) -
Build # 10758 - Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/10758/
Java: 64bit/jdk1.8.0_20-ea-b21 -XX:-UseCompressedOops -XX:+UseParallelGC
1 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch
Error Message:
shard1 is not consistent. Got 129 from http://127.0.0.1:44393/collection1lastClient and got 134 from http://127.0.0.1:47888/collection1
Stack Trace:
java.lang.AssertionError: shard1 is not consistent. Got 129 from http://127.0.0.1:44393/collection1lastClient and got 134 from http://127.0.0.1:47888/collection1
at __randomizedtesting.SeedInfo.seed([C267B49FCD0F1326:43813A87BA50731A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1132)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1111)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.doTest(ChaosMonkeySafeLeaderTest.java:150)
at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
Build Log:
[...truncated 10960 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/init-core-data-001
[junit4] 2> 201393 T849 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 201394 T849 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4] 2> 201398 T849 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 201398 T849 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 1> client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 201399 T850 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 201499 T849 oasc.ZkTestServer.run start zk server on port:43236
[junit4] 2> 201500 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 201502 T856 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4fe1d064 name:ZooKeeperConnection Watcher:127.0.0.1:43236 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 201502 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 201502 T849 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 201505 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 201507 T858 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a87f819 name:ZooKeeperConnection Watcher:127.0.0.1:43236/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 201507 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 201507 T849 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 201509 T849 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 201511 T849 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 201512 T849 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 201513 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 201514 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 201516 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 201516 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 201522 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 201523 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 201526 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 201526 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 201528 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 201528 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 201530 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 201530 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 201531 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 201532 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 201533 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 201534 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 201535 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 201536 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 201537 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 201537 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 201539 T849 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 201539 T849 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 201901 T849 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 201909 T849 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39107
[junit4] 2> 201910 T849 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 201910 T849 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 201910 T849 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002
[junit4] 2> 201911 T849 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002/'
[junit4] 2> 201925 T849 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002/solr.xml
[junit4] 2> 201936 T849 oasc.CoreContainer.<init> New CoreContainer 1090089641
[junit4] 2> 201936 T849 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002/]
[junit4] 2> 201937 T849 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 201937 T849 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 201937 T849 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 201938 T849 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 201938 T849 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 201938 T849 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 201939 T849 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 201939 T849 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 201939 T849 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 201940 T849 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 201941 T849 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 201941 T849 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 201941 T849 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43236/solr
[junit4] 2> 201942 T849 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 201943 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 201945 T869 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58c954b6 name:ZooKeeperConnection Watcher:127.0.0.1:43236 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 201945 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 201947 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 201953 T871 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d4a8f1f name:ZooKeeperConnection Watcher:127.0.0.1:43236/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 201953 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 201955 T849 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 201957 T849 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 201959 T849 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 201960 T849 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 201962 T849 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 201964 T849 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 201965 T849 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39107_
[junit4] 2> 201966 T849 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39107_
[junit4] 2> 201967 T849 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 201969 T849 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 201971 T849 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39107_
[junit4] 2> 201971 T849 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 201973 T849 oasc.Overseer.start Overseer (id=92066953057665027-127.0.0.1:39107_-n_0000000000) starting
[junit4] 2> 201975 T849 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 201981 T849 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 201981 T873 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 201984 T849 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 201986 T849 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 201988 T872 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 201990 T874 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 201990 T874 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 201991 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 201991 T874 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 201992 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 201992 T872 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:39107",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39107_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 201992 T872 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 201992 T872 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 201993 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 201994 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 202991 T874 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 202992 T874 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 202992 T874 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 202993 T874 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 202993 T874 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 202993 T874 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002/collection1/'
[junit4] 2> 202994 T874 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 202994 T874 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 202995 T874 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 203018 T874 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 203029 T874 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 203030 T874 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 203035 T874 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 203139 T874 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 203141 T874 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 203142 T874 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 203145 T874 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 203173 T874 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 203173 T874 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 203184 T874 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/control/data/
[junit4] 2> 203185 T874 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ebd6197
[junit4] 2> 203185 T874 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/control/data
[junit4] 2> 203186 T874 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/control/data/index/
[junit4] 2> 203186 T874 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 203187 T874 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/control/data/index
[junit4] 2> 203188 T874 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=64.0556640625, floorSegmentMB=0.740234375, forceMergeDeletesPctAllowed=11.878602682924864, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 203195 T874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 203195 T874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 203201 T874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 203201 T874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 203202 T874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 203202 T874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 203202 T874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 203202 T874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 203203 T874 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 203203 T874 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 203203 T874 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 203203 T874 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 203204 T874 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 203204 T874 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 203204 T874 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 203204 T874 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 203204 T874 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 203205 T874 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 203205 T874 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 203207 T874 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 203208 T874 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 203208 T874 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 203209 T874 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=82.6162109375, floorSegmentMB=1.125, forceMergeDeletesPctAllowed=13.168835322474951, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 203210 T874 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/control/data/index,segFN=segments_1,generation=1}
[junit4] 2> 203210 T874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 203211 T874 oass.SolrIndexSearcher.<init> Opening Searcher@17d89782[collection1] main
[junit4] 2> 203211 T874 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 203211 T874 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 203212 T874 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 203212 T874 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 203212 T874 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 203212 T874 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 203213 T874 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 203213 T874 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 203213 T874 oascc.SolrZkClient.makePath makePath: /configs/conf1/_rest_managed.json
[junit4] 2> 203215 T874 oasr.ManagedResourceStorage$ZooKeeperStorageIO$1.close Wrote 38 bytes to new znode /configs/conf1/_rest_managed.json
[junit4] 2> 203215 T874 oasr.ManagedResourceStorage$JsonStorage.store Saved JSON object to path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 203215 T874 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 203218 T875 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17d89782[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 203218 T874 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 203219 T878 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39107 collection:control_collection shard:shard1
[junit4] 2> 203219 T849 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 203219 T878 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 203219 T849 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 203221 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 203223 T880 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1fc5ba59 name:ZooKeeperConnection Watcher:127.0.0.1:43236/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 203224 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 203224 T878 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 203225 T849 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 203226 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 203226 T878 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 203227 T878 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C133 name=collection1 org.apache.solr.core.SolrCore@990821e url=http://127.0.0.1:39107/collection1 node=127.0.0.1:39107_ C133_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:39107, core=collection1, node_name=127.0.0.1:39107_}
[junit4] 2> 203227 T878 C133 P39107 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39107/collection1/
[junit4] 2> 203227 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 203227 T849 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 203228 T878 C133 P39107 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 203228 T878 C133 P39107 oasc.SyncStrategy.syncToMe http://127.0.0.1:39107/collection1/ has no replicas
[junit4] 2> 203229 T878 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39107/collection1/ shard1
[junit4] 2> 203229 T878 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 203234 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 203236 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 203337 T880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 203337 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 203385 T878 oasc.ZkController.register We are http://127.0.0.1:39107/collection1/ and leader is http://127.0.0.1:39107/collection1/
[junit4] 2> 203385 T878 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39107
[junit4] 2> 203385 T878 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 203385 T878 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 203386 T878 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 203386 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 203387 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 203387 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 203387 T878 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 203388 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 203396 T872 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:39107",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:39107_",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 203397 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 203498 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 203498 T880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 203575 T849 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 203576 T849 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 203578 T849 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:52142
[junit4] 2> 203579 T849 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 203580 T849 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 203580 T849 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003
[junit4] 2> 203581 T849 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003/'
[junit4] 2> 203598 T849 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003/solr.xml
[junit4] 2> 203608 T849 oasc.CoreContainer.<init> New CoreContainer 2033525747
[junit4] 2> 203609 T849 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003/]
[junit4] 2> 203609 T849 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 203610 T849 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 203610 T849 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 203610 T849 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 203611 T849 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 203611 T849 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 203611 T849 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 203611 T849 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 203612 T849 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 203612 T849 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 203613 T849 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 203613 T849 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 203614 T849 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43236/solr
[junit4] 2> 203614 T849 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 203615 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 203617 T891 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b8a5a81 name:ZooKeeperConnection Watcher:127.0.0.1:43236 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 203617 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 203619 T851 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x1471667f08e0005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 203619 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 203625 T893 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@229055f7 name:ZooKeeperConnection Watcher:127.0.0.1:43236/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 203625 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 203630 T849 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 204633 T849 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52142_
[junit4] 2> 204634 T849 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52142_
[junit4] 2> 204639 T894 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 204640 T894 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 204640 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 204640 T894 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 204640 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 204641 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 204641 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 204642 T872 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:52142",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:52142_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 204642 T872 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2, shard3]
[junit4] 2> 204642 T872 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 204643 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 204745 T880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 204745 T893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 204745 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 205641 T894 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 205641 T894 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 205641 T894 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 205641 T894 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 205642 T894 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 205642 T894 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003/collection1/'
[junit4] 2> 205643 T894 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 205643 T894 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 205643 T894 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 205660 T894 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 205669 T894 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 205670 T894 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 205674 T894 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 205754 T894 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 205755 T894 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 205756 T894 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 205758 T894 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 205770 T894 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 205770 T894 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 205770 T894 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty1/
[junit4] 2> 205770 T894 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ebd6197
[junit4] 2> 205771 T894 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty1
[junit4] 2> 205771 T894 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty1/index/
[junit4] 2> 205771 T894 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 205772 T894 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty1/index
[junit4] 2> 205772 T894 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=64.0556640625, floorSegmentMB=0.740234375, forceMergeDeletesPctAllowed=11.878602682924864, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 205782 T894 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 205783 T894 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 205785 T894 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 205785 T894 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 205786 T894 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 205786 T894 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 205786 T894 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 205786 T894 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 205786 T894 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 205786 T894 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 205786 T894 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 205787 T894 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 205787 T894 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 205787 T894 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 205787 T894 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 205788 T894 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 205788 T894 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 205788 T894 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 205788 T894 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 205790 T894 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 205791 T894 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 205791 T894 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 205792 T894 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=82.6162109375, floorSegmentMB=1.125, forceMergeDeletesPctAllowed=13.168835322474951, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 205793 T894 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty1/index,segFN=segments_1,generation=1}
[junit4] 2> 205793 T894 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 205793 T894 oass.SolrIndexSearcher.<init> Opening Searcher@5c1ce2c7[collection1] main
[junit4] 2> 205794 T894 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 205794 T894 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 205795 T894 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 205795 T894 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 205795 T894 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 205795 T894 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 205796 T894 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 205796 T894 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 205796 T894 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 205796 T894 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 205798 T895 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5c1ce2c7[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 205799 T894 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 205800 T898 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:52142 collection:collection1 shard:shard2
[junit4] 2> 205800 T849 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 205800 T849 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 205800 T898 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 205803 T898 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 205804 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205804 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205805 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205805 T898 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 205805 T898 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C134 name=collection1 org.apache.solr.core.SolrCore@5a1c6940 url=http://127.0.0.1:52142/collection1 node=127.0.0.1:52142_ C134_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:52142, core=collection1, node_name=127.0.0.1:52142_}
[junit4] 2> 205805 T898 C134 P52142 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:52142/collection1/
[junit4] 2> 205805 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 205805 T898 C134 P52142 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 205806 T898 C134 P52142 oasc.SyncStrategy.syncToMe http://127.0.0.1:52142/collection1/ has no replicas
[junit4] 2> 205806 T898 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:52142/collection1/ shard2
[junit4] 2> 205806 T898 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 205807 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205808 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205810 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205911 T880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 205911 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 205911 T893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 205960 T898 oasc.ZkController.register We are http://127.0.0.1:52142/collection1/ and leader is http://127.0.0.1:52142/collection1/
[junit4] 2> 205960 T898 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:52142
[junit4] 2> 205960 T898 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 205960 T898 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 205960 T898 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 205961 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205962 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205962 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 205962 T898 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 205963 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 205963 T872 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:52142",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:52142_",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 205965 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 206066 T880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 206066 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 206066 T893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 206098 T849 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 206099 T849 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 206101 T849 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50386
[junit4] 2> 206102 T849 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 206102 T849 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 206102 T849 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004
[junit4] 2> 206102 T849 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004/'
[junit4] 2> 206118 T849 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004/solr.xml
[junit4] 2> 206125 T849 oasc.CoreContainer.<init> New CoreContainer 1448814016
[junit4] 2> 206126 T849 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004/]
[junit4] 2> 206126 T849 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 206126 T849 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 206127 T849 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 206127 T849 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 206127 T849 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 206127 T849 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 206127 T849 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 206128 T849 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 206128 T849 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 206128 T849 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 206128 T849 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 206129 T849 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 206129 T849 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43236/solr
[junit4] 2> 206129 T849 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 206130 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 206132 T909 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c5baec1 name:ZooKeeperConnection Watcher:127.0.0.1:43236 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 206132 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 206138 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 206139 T911 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5098f925 name:ZooKeeperConnection Watcher:127.0.0.1:43236/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 206139 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 206143 T849 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 207145 T849 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50386_
[junit4] 2> 207146 T849 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50386_
[junit4] 2> 207152 T912 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 207152 T912 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 207153 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 207153 T912 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 207153 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 207153 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 207154 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 207154 T872 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:50386",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50386_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 207154 T872 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 207154 T872 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4] 2> 207155 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 207257 T880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 207257 T911 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 207257 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 207257 T893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208153 T912 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 208153 T912 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 208154 T912 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 208154 T912 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 208155 T912 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 208155 T912 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004/collection1/'
[junit4] 2> 208156 T912 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 208156 T912 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 208156 T912 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 208187 T912 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_5_0
[junit4] 2> 208199 T912 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 208200 T912 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 208205 T912 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4] 2> 208290 T912 oass.IndexSchema.readSchema default search field in schema is text
[junit4] 2> 208291 T912 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 208292 T912 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 208298 T912 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4] 2> 208309 T912 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 208310 T912 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4] 2> 208310 T912 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty2/
[junit4] 2> 208310 T912 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ebd6197
[junit4] 2> 208311 T912 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty2
[junit4] 2> 208311 T912 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty2/index/
[junit4] 2> 208312 T912 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 208312 T912 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty2/index
[junit4] 2> 208313 T912 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=46, maxMergeAtOnceExplicit=10, maxMergedSegmentMB=64.0556640625, floorSegmentMB=0.740234375, forceMergeDeletesPctAllowed=11.878602682924864, segmentsPerTier=20.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 208325 T912 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 208326 T912 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 208328 T912 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
[junit4] 2> 208328 T912 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
[junit4] 2> 208328 T912 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4] 2> 208329 T912 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
[junit4] 2> 208329 T912 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4] 2> 208329 T912 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
[junit4] 2> 208329 T912 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 208329 T912 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4] 2> 208329 T912 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 208330 T912 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4] 2> 208330 T912 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 208330 T912 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 208330 T912 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 208331 T912 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 208331 T912 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4] 2> 208331 T912 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
[junit4] 2> 208331 T912 oasc.RequestHandlers.initHandlersFromConfig created /admin/fileedit: solr.admin.EditFileRequestHandler
[junit4] 2> 208333 T912 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 208334 T912 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 208334 T912 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 208338 T912 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=24, maxMergeAtOnceExplicit=44, maxMergedSegmentMB=82.6162109375, floorSegmentMB=1.125, forceMergeDeletesPctAllowed=13.168835322474951, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 208339 T912 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty2/index,segFN=segments_1,generation=1}
[junit4] 2> 208339 T912 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 208339 T912 oass.SolrIndexSearcher.<init> Opening Searcher@5c968266[collection1] main
[junit4] 2> 208339 T912 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 208340 T912 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 208340 T912 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 208340 T912 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 208341 T912 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 208341 T912 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 208341 T912 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream Read 38 bytes from znode /configs/conf1/_rest_managed.json
[junit4] 2> 208342 T912 oasr.ManagedResourceStorage.load Loaded LinkedHashMap at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 208342 T912 oasr.ManagedResource.reloadFromStorage Loaded initArgs {} for /rest/managed
[junit4] 2> 208342 T912 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 208344 T913 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5c968266[collection1] main{UninvertingDirectoryReader()}
[junit4] 2> 208346 T912 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 208347 T916 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50386 collection:collection1 shard:shard3
[junit4] 2> 208347 T849 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4] 2> 208348 T849 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 208350 T916 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4] 2> 208357 T916 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4] 2> 208358 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208358 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208359 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208359 T916 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 208359 T916 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C135 name=collection1 org.apache.solr.core.SolrCore@7b3ede0f url=http://127.0.0.1:50386/collection1 node=127.0.0.1:50386_ C135_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:50386, core=collection1, node_name=127.0.0.1:50386_}
[junit4] 2> 208359 T916 C135 P50386 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50386/collection1/
[junit4] 2> 208359 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 208360 T916 C135 P50386 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 208360 T916 C135 P50386 oasc.SyncStrategy.syncToMe http://127.0.0.1:50386/collection1/ has no replicas
[junit4] 2> 208360 T916 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50386/collection1/ shard3
[junit4] 2> 208360 T916 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4] 2> 208361 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208363 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208365 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208466 T911 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208467 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208466 T893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208466 T880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208514 T916 oasc.ZkController.register We are http://127.0.0.1:50386/collection1/ and leader is http://127.0.0.1:50386/collection1/
[junit4] 2> 208514 T916 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50386
[junit4] 2> 208514 T916 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 208514 T916 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 208515 T916 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 208516 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208516 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208516 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208516 T916 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 208517 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 208517 T872 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:50386",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:50386_",
[junit4] 2> "shard":"shard3",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":"core_node2"}
[junit4] 2> 208519 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 208574 T849 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 208575 T849 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 208576 T849 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:41023
[junit4] 2> 208577 T849 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 208577 T849 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 208577 T849 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-005
[junit4] 2> 208578 T849 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-005/'
[junit4] 2> 208588 T849 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-005/solr.xml
[junit4] 2> 208595 T849 oasc.CoreContainer.<init> New CoreContainer 251250139
[junit4] 2> 208595 T849 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-005/]
[junit4] 2> 208596 T849 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 208596 T849 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 208596 T849 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 208597 T849 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 208597 T849 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 208597 T849 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 208597 T849 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 208597 T849 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 208598 T849 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 208598 T849 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 208598 T849 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 208598 T849 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 208599 T849 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43236/solr
[junit4] 2> 208599 T849 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 208600 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 208601 T927 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@694b601a name:ZooKeeperConnection Watcher:127.0.0.1:43236 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 208602 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 208604 T849 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 208605 T929 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3059ad1a name:ZooKeeperConnection Watcher:127.0.0.1:43236/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 208606 T849 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 208609 T849 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 208620 T911 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208620 T880 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208620 T893 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208620 T929 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 208620 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 209611 T849 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41023_
[junit4] 2> 209612 T849 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:41023_
[junit4] 2> 209617 T930 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 209618 T930 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 209618 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 209618 T930 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 209618 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 209621 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 209622 T872 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 209622 T872 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:41023",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:41023_",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"3",
[junit4] 2> "core_node_name":null}
[junit4] 2> 209623 T872 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4] 2> 209623 T872 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 209624 T871 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 209725 T911 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4] 2> 209726 T871 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConne
[...truncated too long message...]
:/clusterstate.json, has occurred - updating... (live nodes size: 11)
[junit4] 2> 310961 T1181 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 311047 T1233 C513 P47888 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 311048 T1233 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 311048 T1233 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@55af4817
[junit4] 2> 311049 T1233 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=22,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=198,cumulative_deletesById=128,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=16877,transaction_logs_total_number=1}
[junit4] 2> 311050 T1233 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 311050 T1233 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 311050 T1233 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 311057 T1233 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 311058 T1233 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 311058 T1233 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty12/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty12/index;done=false>>]
[junit4] 2> 311059 T1233 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty12/index
[junit4] 2> 311059 T1233 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty12 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty12;done=false>>]
[junit4] 2> 311059 T1233 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty12
[junit4] 2> 311059 T1233 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> 311060 T1233 oasc.LeaderElector$ElectionWatcher.process WARN org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:260)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:257)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:94)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:55)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:358)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
[junit4] 2>
[junit4] 2> 311504 T1338 C521 P52640 oasc.SyncStrategy.sync WARN Closed, skipping sync up.
[junit4] 2> 311504 T1338 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4] 2> 311504 T1338 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@4584b108
[junit4] 2> 311506 T1338 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=22,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=68,cumulative_deletesById=36,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=7241,transaction_logs_total_number=2}
[junit4] 2> 311506 T1338 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 311506 T1338 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 311506 T1338 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 311509 T1338 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 311509 T1338 oasc.CachingDirectoryFactory.close Closing StandardDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 311509 T1338 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty11/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty11/index;done=false>>]
[junit4] 2> 311510 T1338 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty11/index
[junit4] 2> 311510 T1338 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty11 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty11;done=false>>]
[junit4] 2> 311510 T1338 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001/tempDir-001/jetty11
[junit4] 2> 311510 T1338 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4] 2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./temp/solr.cloud.ChaosMonkeySafeLeaderTest-C267B49FCD0F1326-001
[junit4] 2> NOTE: test params are: codec=Lucene45, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=el_GR, timezone=US/Hawaii
[junit4] 2> NOTE: Linux 3.8.0-42-generic amd64/Oracle Corporation 1.8.0_20-ea (64-bit)/cpus=8,threads=1,free=104070840,total=330825728
[junit4] 2> NOTE: All tests run in this JVM: [TestRestManager, TestCSVResponseWriter, TestImplicitCoreProperties, ConvertedLegacyTest, TestDistributedGrouping, TestIndexSearcher, TestBadConfig, CollectionsAPIDistributedZkTest, TestIndexingPerformance, SystemInfoHandlerTest, DefaultValueUpdateProcessorTest, TestZkChroot, SOLR749Test, ParsingFieldUpdateProcessorsTest, HdfsLockFactoryTest, DistanceFunctionTest, AtomicUpdatesTest, DistributedExpandComponentTest, TestQuerySenderNoQuery, AddBlockUpdateTest, SolrPluginUtilsTest, TestSolrQueryParser, TestOverriddenPrefixQueryForCustomFieldType, MergeStrategyTest, QueryResultKeyTest, TestStressLucene, ChaosMonkeySafeLeaderTest]
[junit4] Completed on J1 in 110.87s, 1 test, 1 failure <<< FAILURES!
[...truncated 1097 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1298: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:922: There were test failures: 417 suites, 1708 tests, 1 failure, 38 ignored (17 assumptions)
Total time: 67 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_20-ea-b21 -XX:-UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any